Welcome Guest.   Make a donation to an author on the site December 21, 2014, 09:09:42 AM  *

Please login or register.
Or did you miss your validation email?


Login with username and password (forgot your password?)
Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.


You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.
 
The N.A.N.Y. Challenge 2010! Download 24 custom programs!
   
   Forum Home   Thread Marks Chat! Downloads Search Login Register  
Pages: [1] 2 3 Next   Go Down
  Reply  |  New Topic  |  Print  
Author Topic: File synchronization: moving away from incremental backup (HELP!)  (Read 27836 times)
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« on: November 08, 2007, 09:49:02 AM »

At the museum, I'm currently using a backup where each client machine does incremental backups to a file-server, using Acronis TrueIimage. For various reasons (including bugs & quirks with TI), this is no longer desirable. In fact, I don't want to do traditional backups from the client machines to the server.

We're going to outsource backups to a secure data facility, after all having backups on the server is no good if the museum burns down. So, the new scheme I want to run is client machines *synchronizing* files to the server, and the server being backed up incrementally to the remote facility.

So, what good synchronizing programs are out there? I have a few requirements:

  • MUST BE ROBUST!
  • Unobtrusive, must run entirely in the background
  • Must be able to handle open files in a sensible way
  • Preferably light on resources
  • Preferably something that auto-detects changes, (no full harddrive scans)

EDIT: some clarifications, I do not need the following:

  • Backup functionality - that will be handled by another piece of software
  • Revisions/Incremental sync - I just want "plain and simple" file sync
  • Compression - again, handled by backup software on the server
« Last Edit: November 09, 2007, 07:26:21 AM by f0dder » Logged

- carpe noctem
Ralf Maximus
Supporting Member
**
Posts: 927



View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #1 on: November 08, 2007, 10:04:32 AM »

SyncBack.  Perhaps the SE version, but the freeware edition may be sufficient for your needs.

It does all the things you request and even throws in throttling and FTP synchronization too.
Logged
tomos
Charter Member
***
Posts: 8,730



see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #2 on: November 08, 2007, 10:19:52 AM »

have a look at SyncBackSE vs. SuperFlexible thread
Logged

Tom
yksyks
Supporting Member
**
Posts: 475


see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #3 on: November 08, 2007, 10:50:08 AM »

Anyone tried Mozy? Might be a proper solution for this situation.
Logged
czb
Honorary Member
**
Posts: 335



see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #4 on: November 08, 2007, 01:14:18 PM »

I use drivehq and I am completely satisfied Wink

EDIT: it has got synchronization, incremental backup etc. 1GB is free  Kiss
Logged

My open-source online piano game: http://musicope.com/
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #5 on: November 09, 2007, 07:24:38 AM »

Ralf Maximus and tomos: both SyncBack and SuperFlexible seem to be run by a scheduler, instead of doing transparent mirroring... while this could be enough if it's 100% invisible in the background, it still requires checking each file in the "monitored" hierarchy for changes, instead of using filesystem event notifications?

Reading the versus-thread was a good idea though, I will be checking into FileHasmter, MirrorFolder and Second Copy, as those three might be closer to what I need.

Remember, I just need to sync/mirror userdata to \\server\backup\<username>, I don't need any incremental backups/versioning, ftp support, compression etc - all that will be handled by another piece of software responsible for backing up the server.

yksyks: thanks for the link, but we've already settled on a remote backup host (we're moving to fiber internet, and that company also offers remote backup). We need around 100-200gigs of space (and more every year), so transferring the baseline backup will be done by techs from the company, instead of taking a lot of time with a limited 5mbit upload smiley
Logged

- carpe noctem
tomos
Charter Member
***
Posts: 8,730



see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #6 on: November 09, 2007, 08:30:10 AM »

[SFFS] still requires checking each file in the "monitored" hierarchy for changes, instead of using filesystem event notifications?

I believe so
Logged

Tom
nosh
Supporting Member
**
Posts: 1,397


View Profile Give some DonationCredits to this forum member
« Reply #7 on: November 09, 2007, 09:18:34 AM »

I'm pretty certain Second Copy checks each file too coz they warn you of slowdowns when you enable realtime mirroring.
Logged
Ralf Maximus
Supporting Member
**
Posts: 927



View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #8 on: November 09, 2007, 10:06:33 AM »

Quote
Ralf Maximus and tomos: both SyncBack and SuperFlexible seem to be run by a scheduler, instead of doing transparent mirroring... while this could be enough if it's 100% invisible in the background, it still requires checking each file in the "monitored" hierarchy for changes, instead of using filesystem event notifications?

Aye, tis true.  However I am extremely pleased at how fast SyncBack performs its checks.  I routinely synchronize network folders containing dozens of gigabytes, and if there are no changes it takes less than a minute.  Massive changes, of course, incur the time required to copy the files -- standard network traffic, which could take many minutes.

If the file's in use (or becomes in-use) during a copy, SyncBack notes the event in the log and moves on without hassling the user.  This is a configuration setting.

As far as background processing... I believe it can be configured to run silently, with no visible GUI.  I often call SyncBack from .bat files using its command-line interface, and those events generate nothing visible, not even a tray icon.  And it throttles its CPU demand to prevent taking over the user's descktop experience; aside from a chattering disk drive they won't even know it's running.

Not bad for a freebie.
Logged
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #9 on: November 10, 2007, 05:27:36 AM »

I've checked out MirrorFolder a bit, and it seems very very very promising.

As far as I've been able to tell, the best the other applications can offer is detecting modified files via filesystem events - this has two problems. One is that Windows' fsevents only tell you that "there's a change somewhere in the tree you're monitoring", not WHAT has changed - which means you have to scan the entire tree upon notification (unless you resort to dirty tricks like reading the NTFS journal). The other problem is that you have to synchronize the entire file, even if only part of it has changed (probably not too big a problem for my uses, but still...)

MirrorFolder is cool, though. It can install a filter driver that monitors changes, so only the changes are synced - ie., changing 10 bytes in a 10gig file only pushes those 10 bytes to your sync destination. And it even works with network shares (couldn't find a way to set user credentials, but mapping at network drive worked like a charm). The driver is 73kb and the server is 118kb, so it's not exactly bloatware either.

I will have to do some further testing of this, but MirrorFolder seems like a natural winner for my purposes, if it doesn't impact performance too much, and (of course!) is stable. But I expect the filter driver approach to have pretty decent performance. The only nag I have about MF is that they call the filter drive method for "RAID-1", which is wrong and misleading and doesn't give the technique enough credit smiley

Still, if anybody has other suggestions, keep 'em coming!
Logged

- carpe noctem
tomos
Charter Member
***
Posts: 8,730



see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #10 on: November 10, 2007, 08:31:23 AM »

from the surfulater archives smiley

IBM's Tivoli Continuous Data Protection http://www-306.ibm.com/so...ntinuous-data-protection/.  Note:  I DO work for IBM, so I get this software for free vs the $36 USD normal price.  If I had to pay for the Tivoli software, I'd likely just use the free software that came with my Diskstation....not because Tivoli software is lacking.....my impression is it is top notch but I am just cheap and typically lean towards freeware if available.  Be it Tivoli or Diskstation software....they both use the same approach....a real time data copy based on a set of rules.  So I tell it what files to copy (be it filename matches, filetype matches or folder matches) and as soon as I save a file which meets the criteria, it copies it to a backup.  It also can save multiple versions and target files require no special software to open.....meaning the files are not imaged but real file copies.  I prefer this real-time backup as opposed to a scheduled backup
Logged

Tom
Ralf Maximus
Supporting Member
**
Posts: 927



View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #11 on: November 10, 2007, 11:31:37 AM »

I've checked out MirrorFolder a bit, and it seems very very very promising.

Wow, MirrorFolder looks insanely cool.   And for $39 US it seems reasonable.  I assume buying a single license allows one to synchronize to another workstation/server?  The FAQ doesn't say.  My excitement would diminish a bit if I have to buy a license for the target PC/server also.

This is a heavy duty piece of low-level engineering, BTW.  Installing as a volume filter means if MirrorFolder isn't 100% solid 100% of the time you'll become friends with the BSOD.   I learned this with SuperCache-II, which is also a volume filter.  SC-II is rock solid, except for one feature which, when activated, makes file activity on my system a game of random chance.  So I have that aspect of the program turned off and everything's fine.

Not saying volume filter technology is evil -- far from it -- but unlike application level code, if it tanks it takes the whole system down with it.  It does suggest that it be tested thouroughly in the target environment before making a decision.
Logged
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #12 on: November 10, 2007, 11:41:27 AM »

Wow, MirrorFolder looks insanely cool.   And for $39 US it seems reasonable.  I assume buying a single license allows one to synchronize to another workstation/server?  The FAQ doesn't say.  My excitement would diminish a bit if I have to buy a license for the target PC/server also.
Can't see why it should require a license, you're only installing on the client machine... and the app does seem to be UNC path aware (I mapped drive X to a server path, but the GUI shows the UNC path drive X is mapped to. I did the drive mapping because I couldn't find a way to enter user credentials, only straight UNC paths).

This is a heavy duty piece of low-level engineering, BTW.  Installing as a volume filter means if MirrorFolder isn't 100% solid 100% of the time you'll become friends with the BSOD.   I learned this with SuperCache-II, which is also a volume filter.
Yup - that's why I installed under vmware and not on my main box. Haven't stress-tested it, but it does work. And there's both 32- and 64-bit versions available, which gives me at least some confidence.

It does suggest that it be tested thouroughly in the target environment before making a decision.
That too - once I've done preliminary testing here, I'll have to install it on one of the museum's PC's and let it run in trial mode for a couple of weeks, and spot problems. Hope it doesn't interfere with things like antivirus etc.

But it sure does look promising!
Logged

- carpe noctem
Armando
Charter Member
***
Posts: 2,683



see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #13 on: November 10, 2007, 04:22:12 PM »

I really like the "filter driver" feature. f00der : does that seem to translate into real performance gain (mirroring time) ?
Logged

"I suppose it can be said that I'm an absent-minded driver. It's true that I've driven through a number of red lights on occasion, but on the other hand, I've stopped at a lot of green ones but never gotten credit for it."
Glenn Gould
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #14 on: November 10, 2007, 04:48:50 PM »

I really like the "filter driver" feature. f00der : does that seem to translate into real performance gain (mirroring time) ?
I haven't tested it extensively yet, and I'd say it depends on what you're doing (although I can't imagine it would ever be slower than other methods). If you have some large files, it definitely should be faster, since only the changed portions are going to be synced. And not having to scan a potentially deep folder tree for changes on a fs-change event notify can also save time.
Logged

- carpe noctem
Armando
Charter Member
***
Posts: 2,683



see users location on a map View Profile Give some DonationCredits to this forum member
« Reply #15 on: November 10, 2007, 04:55:03 PM »

Thanks! I'm asking because I didn't find that SFFS's delta backup technology accelerated my backups by that much, if any acceleration.

not having to scan a potentially deep folder tree for changes on a fs-change event notify can also save time

Definitely. Yes, there's that too...
Logged

"I suppose it can be said that I'm an absent-minded driver. It's true that I've driven through a number of red lights on occasion, but on the other hand, I've stopped at a lot of green ones but never gotten credit for it."
Glenn Gould
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #16 on: November 10, 2007, 05:05:52 PM »

SFFS delta backup probably has to read through the entire file to check which range(s) have changed... that's take some time. But I haven't read up on how it does it, so it's just a guess smiley
Logged

- carpe noctem
tomos
Charter Member
***
Posts: 8,730



see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #17 on: November 11, 2007, 06:34:46 AM »

Thanks! I'm asking because I didn't find that SFFS's delta backup technology accelerated my backups by that much, if any acceleration.

on my last computer I was using a usb1.1 connection sometimes -
partial file update made a BIG difference there smiley
but also notice a speed difference with usb2

but I just have it running in the background anyways &
while I synch some big files, there's not many of them & I've never bothered timing them..
Logged

Tom
AEngineer
Supporting Member
**
Posts: 42


View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #18 on: November 20, 2007, 07:28:01 AM »

Here's a quick plug for a MS product (bought) - Foldershare.  I've been using it for several years to sync folders on five machines(!) at different locations, desktop and laptops both.  It does sometimes produce extra copies of files if there have been simultaneous edits, quite easily found and deleted, but I've never lost anything.
Free
Limit of 10G per sync
Limit of about 10 simultaneous syncs
It will sync Macs and PC's but not (surprise?) linux.
I'd gladly pay some for a "Pro" version with more capacity but they seem happy to let it languish.  There have been no changes since MS bought it.
Logged

Jim Mitchell
MerleOne
Supporting Member
**
Posts: 893


4D thinking

see users location on a map View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #19 on: November 20, 2007, 07:44:43 AM »

NTI Shadows seem to do what you are looking for :
http://www.ntius.com/shadow.asp
Logged

.merle1.
Dr-Leech
Member
**
Posts: 151


Psychedelic Coder

View Profile WWW Give some DonationCredits to this forum member
« Reply #20 on: November 20, 2007, 07:52:21 AM »

Cobian Backup is a really good alternative. And free.
Logged

f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #21 on: November 20, 2007, 07:59:01 AM »

NTI Shadows seem to do what you are looking for :
http://www.ntius.com/shadow.asp
Does NTI do the filter driver thing?
Logged

- carpe noctem
MerleOne
Supporting Member
**
Posts: 893


4D thinking

see users location on a map View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #22 on: November 20, 2007, 04:00:45 PM »

NTI Shadows seem to do what you are looking for :
http://www.ntius.com/shadow.asp
Does NTI do the filter driver thing?


If you mean by that partial file backup, I am almost sure it is not the case. Previous version of NTI shadow were pretty basic, I don't know the full details about this release but I would be surprised if it were the case.  Anyway, there is probably a time limited demo which could help you find out.  Sorry not to have more info !
Logged

.merle1.
plutchok
Supporting Member
**
Posts: 7

see users location on a map View Profile WWW Give some DonationCredits to this forum member
« Reply #23 on: November 22, 2007, 12:57:58 PM »

MS SyncToy will do what you want, but awkwardly. GoodSync is too limited. I use AllWay Sync'n'Go and am satisfied. You can configure it to work just as you said you want it to. there's also a nice USB version. See my review in post #27 of my utilities blog, http://jonathanstoolbar.blogspot.com . Many others have also recommended Karen's File Replicator (http://www.karenware.com/powertools/powertools.asp), but I haven't tried it.
Logged
MerleOne
Supporting Member
**
Posts: 893


4D thinking

see users location on a map View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #24 on: November 22, 2007, 02:09:12 PM »

SFFS delta backup probably has to read through the entire file to check which range(s) have changed... that's take some time. But I haven't read up on how it does it, so it's just a guess smiley

SFFS cuts the file in smaller parts, performs CRC on a part, same on the target file, and if CRCs differ, this part is updated, and so on.  When target is on a remote server, CRC is performed locally by a Windows service dedicated, so no big data transfer is required.
Logged

.merle1.
Pages: [1] 2 3 Next   Go Up
  Reply  |  New Topic  |  Print  
 
Jump to:  
   Forum Home   Thread Marks Chat! Downloads Search Login Register  

DonationCoder.com | About Us
DonationCoder.com Forum | Powered by SMF
[ Page time: 0.051s | Server load: 0.07 ]