Author Topic: File synchronization: moving away from incremental backup (HELP!) (Read 62990 times)

f0dder · « **on:** November 08, 2007, 09:49 AM »

At the museum, I'm currently using a backup where each client machine does incremental backups to a file-server, using Acronis TrueIimage. For various reasons (including bugs & quirks with TI), this is no longer desirable. In fact, I don't want to do traditional backups from the client machines to the server.

We're going to outsource backups to a secure data facility, after all having backups on the server is no good if the museum burns down. So, the new scheme I want to run is client machines *synchronizing* files to the server, and the server being backed up incrementally to the remote facility.

So, what good synchronizing programs are out there? I have a few requirements:

MUST BE ROBUST!
Unobtrusive, must run entirely in the background
Must be able to handle open files in a sensible way
Preferably light on resources
Preferably something that auto-detects changes, (no full harddrive scans)

EDIT: some clarifications, I do not need the following:

Backup functionality - that will be handled by another piece of software
Revisions/Incremental sync - I just want "plain and simple" file sync
Compression - again, handled by backup software on the server

Ralf Maximus · « **Reply #1 on:** November 08, 2007, 10:04 AM »

SyncBack. Perhaps the SE version, but the freeware edition may be sufficient for your needs.

It does all the things you request and even throws in throttling and FTP synchronization too.

tomos · « **Reply #2 on:** November 08, 2007, 10:19 AM »

have a look at SyncBackSE vs. SuperFlexible thread

yksyks · « **Reply #3 on:** November 08, 2007, 10:50 AM »

Anyone tried Mozy? Might be a proper solution for this situation.

czb · « **Reply #4 on:** November 08, 2007, 01:14 PM »

I use drivehq and I am completely satisfied

EDIT: it has got synchronization, incremental backup etc. 1GB is free

f0dder · « **Reply #5 on:** November 09, 2007, 07:24 AM »

Ralf Maximus and tomos: both SyncBack and SuperFlexible seem to be run by a scheduler, instead of doing transparent mirroring... while this could be enough if it's 100% invisible in the background, it still requires checking each file in the "monitored" hierarchy for changes, instead of using filesystem event notifications?

Reading the versus-thread was a good idea though, I will be checking into FileHasmter, MirrorFolder and Second Copy, as those three might be closer to what I need.

Remember, I just need to sync/mirror userdata to \\server\backup\<username>, I don't need any incremental backups/versioning, ftp support, compression etc - all that will be handled by another piece of software responsible for backing up the server.

yksyks: thanks for the link, but we've already settled on a remote backup host (we're moving to fiber internet, and that company also offers remote backup). We need around 100-200gigs of space (and more every year), so transferring the baseline backup will be done by techs from the company, instead of taking a lot of time with a limited 5mbit upload

tomos · « **Reply #6 on:** November 09, 2007, 08:30 AM »

[SFFS] still requires checking each file in the "monitored" hierarchy for changes, instead of using filesystem event notifications?
-f0dder (November 09, 2007, 07:24 AM)

I believe so

nosh · « **Reply #7 on:** November 09, 2007, 09:18 AM »

I'm pretty certain Second Copy checks each file too coz they warn you of slowdowns when you enable realtime mirroring.

Ralf Maximus · « **Reply #8 on:** November 09, 2007, 10:06 AM »

Ralf Maximus and tomos: both SyncBack and SuperFlexible seem to be run by a scheduler, instead of doing transparent mirroring... while this could be enough if it's 100% invisible in the background, it still requires checking each file in the "monitored" hierarchy for changes, instead of using filesystem event notifications?

Aye, tis true. However I am extremely pleased at how fast SyncBack performs its checks. I routinely synchronize network folders containing dozens of gigabytes, and if there are no changes it takes less than a minute. Massive changes, of course, incur the time required to copy the files -- standard network traffic, which could take many minutes.

If the file's in use (or becomes in-use) during a copy, SyncBack notes the event in the log and moves on without hassling the user. This is a configuration setting.

As far as background processing... I believe it can be configured to run silently, with no visible GUI. I often call SyncBack from .bat files using its command-line interface, and those events generate nothing visible, not even a tray icon. And it throttles its CPU demand to prevent taking over the user's descktop experience; aside from a chattering disk drive they won't even know it's running.

Not bad for a freebie.

f0dder · « **Reply #9 on:** November 10, 2007, 05:27 AM »

I've checked out MirrorFolder a bit, and it seems very very very promising.

As far as I've been able to tell, the best the other applications can offer is detecting modified files via filesystem events - this has two problems. One is that Windows' fsevents only tell you that "there's a change somewhere in the tree you're monitoring", not WHAT has changed - which means you have to scan the entire tree upon notification (unless you resort to dirty tricks like reading the NTFS journal). The other problem is that you have to synchronize the entire file, even if only part of it has changed (probably not too big a problem for my uses, but still...)

MirrorFolder is cool, though. It can install a filter driver that monitors changes, so only the changes are synced - ie., changing 10 bytes in a 10gig file only pushes those 10 bytes to your sync destination. And it even works with network shares (couldn't find a way to set user credentials, but mapping at network drive worked like a charm). The driver is 73kb and the server is 118kb, so it's not exactly bloatware either.

I will have to do some further testing of this, but MirrorFolder seems like a natural winner for my purposes, if it doesn't impact performance too much, and (of course!) is stable. But I expect the filter driver approach to have pretty decent performance. The only nag I have about MF is that they call the filter drive method for "RAID-1", which is wrong and misleading and doesn't give the technique enough credit

Still, if anybody has other suggestions, keep 'em coming!

tomos · « **Reply #10 on:** November 10, 2007, 08:31 AM »

from the surfulater archives

IBM's Tivoli Continuous Data Protection http://www-306.ibm.c...ous-data-protection/. Note: I DO work for IBM, so I get this software for free vs the $36 USD normal price. If I had to pay for the Tivoli software, I'd likely just use the free software that came with my Diskstation....not because Tivoli software is lacking.....my impression is it is top notch but I am just cheap and typically lean towards freeware if available. Be it Tivoli or Diskstation software....they both use the same approach....a real time data copy based on a set of rules. So I tell it what files to copy (be it filename matches, filetype matches or folder matches) and as soon as I save a file which meets the criteria, it copies it to a backup. It also can save multiple versions and target files require no special software to open.....meaning the files are not imaged but real file copies. I prefer this real-time backup as opposed to a scheduled backup
-johnfdeluca (May 11, 2007, 09:53 AM)

Ralf Maximus · « **Reply #11 on:** November 10, 2007, 11:31 AM »

I've checked out MirrorFolder a bit, and it seems very very very promising.
-f0dder (November 10, 2007, 05:27 AM)

Wow, MirrorFolder looks insanely cool. And for $39 US it seems reasonable. I assume buying a single license allows one to synchronize to another workstation/server? The FAQ doesn't say. My excitement would diminish a bit if I have to buy a license for the target PC/server also.

This is a heavy duty piece of low-level engineering, BTW. Installing as a volume filter means if MirrorFolder isn't 100% solid 100% of the time you'll become friends with the BSOD. I learned this with SuperCache-II, which is also a volume filter. SC-II is rock solid, except for one feature which, when activated, makes file activity on my system a game of random chance. So I have that aspect of the program turned off and everything's fine.

Not saying volume filter technology is evil -- far from it -- but unlike application level code, if it tanks it takes the whole system down with it. It does suggest that it be tested thouroughly in the target environment before making a decision.

f0dder · « **Reply #12 on:** November 10, 2007, 11:41 AM »

Wow, MirrorFolder looks insanely cool. And for $39 US it seems reasonable. I assume buying a single license allows one to synchronize to another workstation/server? The FAQ doesn't say. My excitement would diminish a bit if I have to buy a license for the target PC/server also.
-Ralf Maximus (November 10, 2007, 11:31 AM)

Can't see why it should require a license, you're only installing on the client machine... and the app does seem to be UNC path aware (I mapped drive X to a server path, but the GUI shows the UNC path drive X is mapped to. I did the drive mapping because I couldn't find a way to enter user credentials, only straight UNC paths).

This is a heavy duty piece of low-level engineering, BTW. Installing as a volume filter means if MirrorFolder isn't 100% solid 100% of the time you'll become friends with the BSOD. I learned this with SuperCache-II, which is also a volume filter.
-Ralf Maximus (November 10, 2007, 11:31 AM)

Yup - that's why I installed under vmware and not on my main box. Haven't stress-tested it, but it does work. And there's both 32- and 64-bit versions available, which gives me at least some confidence.

It does suggest that it be tested thouroughly in the target environment before making a decision.
-Ralf Maximus (November 10, 2007, 11:31 AM)

That too - once I've done preliminary testing here, I'll have to install it on one of the museum's PC's and let it run in trial mode for a couple of weeks, and spot problems. Hope it doesn't interfere with things like antivirus etc.

But it sure does look promising!

Armando · « **Reply #13 on:** November 10, 2007, 04:22 PM »

I really like the "filter driver" feature. f00der : does that seem to translate into real performance gain (mirroring time) ?

f0dder · « **Reply #14 on:** November 10, 2007, 04:48 PM »

I really like the "filter driver" feature. f00der : does that seem to translate into real performance gain (mirroring time) ?
-Armando (November 10, 2007, 04:22 PM)

I haven't tested it extensively yet, and I'd say it depends on what you're doing (although I can't imagine it would ever be slower than other methods). If you have some large files, it definitely should be faster, since only the changed portions are going to be synced. And not having to scan a potentially deep folder tree for changes on a fs-change event notify can also save time.

Armando · « **Reply #15 on:** November 10, 2007, 04:55 PM »

Thanks! I'm asking because I didn't find that SFFS's delta backup technology accelerated my backups by that much, if any acceleration.

not having to scan a potentially deep folder tree for changes on a fs-change event notify can also save time
-f0dder (November 10, 2007, 04:48 PM)

Definitely. Yes, there's that too...

f0dder · « **Reply #16 on:** November 10, 2007, 05:05 PM »

SFFS delta backup probably has to read through the entire file to check which range(s) have changed... that's take some time. But I haven't read up on how it does it, so it's just a guess

tomos · « **Reply #17 on:** November 11, 2007, 06:34 AM »

Thanks! I'm asking because I didn't find that SFFS's delta backup technology accelerated my backups by that much, if any acceleration.
-Armando (November 10, 2007, 04:55 PM)

on my last computer I was using a usb1.1 connection sometimes -
partial file update made a BIG difference there

but also notice a speed difference with usb2

but I just have it running in the background anyways &
while I synch some big files, there's not many of them & I've never bothered timing them..

AEngineer · « **Reply #18 on:** November 20, 2007, 07:28 AM »

Here's a quick plug for a MS product (bought) - Foldershare. I've been using it for several years to sync folders on five machines(!) at different locations, desktop and laptops both. It does sometimes produce extra copies of files if there have been simultaneous edits, quite easily found and deleted, but I've never lost anything.
Free
Limit of 10G per sync
Limit of about 10 simultaneous syncs
It will sync Macs and PC's but not (surprise?) linux.
I'd gladly pay some for a "Pro" version with more capacity but they seem happy to let it languish. There have been no changes since MS bought it.

MerleOne · « **Reply #19 on:** November 20, 2007, 07:44 AM »

NTI Shadows seem to do what you are looking for :
http://www.ntius.com/shadow.asp

Dr-Leech · « **Reply #20 on:** November 20, 2007, 07:52 AM »

Cobian Backup is a really good alternative. And free.

f0dder · « **Reply #21 on:** November 20, 2007, 07:59 AM »

NTI Shadows seem to do what you are looking for :
http://www.ntius.com/shadow.asp
-MerleOne (November 20, 2007, 07:44 AM)

Does NTI do the filter driver thing?

MerleOne · « **Reply #22 on:** November 20, 2007, 04:00 PM »

NTI Shadows seem to do what you are looking for :
http://www.ntius.com/shadow.asp
-MerleOne (November 20, 2007, 07:44 AM)
Does NTI do the filter driver thing?

-f0dder (November 20, 2007, 07:59 AM)

If you mean by that partial file backup, I am almost sure it is not the case. Previous version of NTI shadow were pretty basic, I don't know the full details about this release but I would be surprised if it were the case. Anyway, there is probably a time limited demo which could help you find out. Sorry not to have more info !

plutchok · « **Reply #23 on:** November 22, 2007, 12:57 PM »

MS SyncToy will do what you want, but awkwardly. GoodSync is too limited. I use AllWay Sync'n'Go and am satisfied. You can configure it to work just as you said you want it to. there's also a nice USB version. See my review in post #27 of my utilities blog, http://jonathanstoolbar.blogspot.com . Many others have also recommended Karen's File Replicator (http://www.karenware...tools/powertools.asp), but I haven't tried it.

MerleOne · « **Reply #24 on:** November 22, 2007, 02:09 PM »

SFFS delta backup probably has to read through the entire file to check which range(s) have changed... that's take some time. But I haven't read up on how it does it, so it's just a guess
-f0dder (November 10, 2007, 05:05 PM)

SFFS cuts the file in smaller parts, performs CRC on a part, same on the target file, and if CRCs differ, this part is updated, and so on. When target is on a remote server, CRC is performed locally by a Windows service dedicated, so no big data transfer is required.