Author Topic: File synchronization: moving away from incremental backup (HELP!) (Read 62978 times)

f0dder · « **Reply #25 on:** November 22, 2007, 02:14 PM »

SFFS delta backup probably has to read through the entire file to check which range(s) have changed... that's take some time. But I haven't read up on how it does it, so it's just a guess
-f0dder (November 10, 2007, 05:05 PM)

SFFS cuts the file in smaller parts, performs CRC on a part, same on the target file, and if CRCs differ, this part is updated, and so on. When target is on a remote server, CRC is performed locally by a Windows service dedicated, so no big data transfer is required.
-MerleOne (November 22, 2007, 02:09 PM)

Smart - but still requires a lot of processing, compared to filter driver approach.

MerleOne · « **Reply #26 on:** November 23, 2007, 12:37 AM »

SFFS delta backup probably has to read through the entire file to check which range(s) have changed... that's take some time. But I haven't read up on how it does it, so it's just a guess
-f0dder (November 10, 2007, 05:05 PM)
SFFS cuts the file in smaller parts, performs CRC on a part, same on the target file, and if CRCs differ, this part is updated, and so on. When target is on a remote server, CRC is performed locally by a Windows service dedicated, so no big data transfer is required.
-MerleOne (November 22, 2007, 02:09 PM)
Smart - but still requires a lot of processing, compared to filter driver approach.
-f0dder (November 22, 2007, 02:14 PM)

Actually it is quite fast and doesn't seem to take that much resources, except maybe HDD access. Some years ago, Laplink used the same technique to update big files across modem links and it was quite effective, and computing power at that time was so much lower. Anyway, the best would be to test it in the real environment and see how it performs.

steeladept · « **Reply #27 on:** November 26, 2007, 09:24 PM »

At work we use Tivoli Storage Manager for this type of thing. I don't know much about it other than it is an effective and transparent way to backup specified files once it is set up. I believe it is free too, though don't quote me on that. For some reason I believe it is an IBM product now though. At any rate, it is definitely something that will handle every requirement you have, as long as the price is right.

Armando · « **Reply #28 on:** November 26, 2007, 10:58 PM »

There were some stuff about Tivoli there :
https://www.donation...40.msg61084#msg61084

Here you can a comparison between the different Tivoli products :

http://www-111.ibm.c...C02=C136879G75391P17

Here are the details for Tivoli Storage Manager :

http://www-306.ibm.c...roducts/storage-mgr/

And the pricing :

https://www-112.ibm....=none&S_CMP=none

Seems interesting... It's even multi-platform : Windows, Linux, HP Unix, Sun Solaris, AIX.

f0dder · « **Reply #29 on:** November 27, 2007, 05:39 AM »

Tivoli has been IBM for, like, forever - at least as long as I can remember (which isn't more than max 5 years though

).

johnfdeluca · « **Reply #30 on:** November 29, 2007, 05:38 PM »

I use Tivoli Continuous Data Protection (again...disclaimer, I work for IBM). Note...this is different than Tivoli Storage Manager. It is $42 and I have been pleased with it. I use it to do insta-synching of files (you might call this mirroring) to a network NAS (which does RAID). It runs in the background, seems to take small resources to run and auto backs up each file as soon as it is saved or updated. I particularly like:

Set it and Forget it - I never really need to check on it. The only time I am reminded that it is running is when it cannot connect to the network drive
Continuously protects versions of files to allow customers choice of recovery points
Destination is not in propriatary format.....same format
Offers to only send the portion of the file that has changed vs. entire file again
Encrypted and/or Compressed options

http://www-306.ibm.c...ous-data-protection/

Free Trial here: http://www-306.ibm.c...nter/storage/cdp.jsp
Flash Demo here: http://www-306.ibm.c...brary/demos/cdp.html

All that said, I'm not sure I'd shell out $42 for it with free options out there but there is something to be said for Big Blue behind the scenes.

General Settings Screenshot

Screenshot - 11_29_2007 , 6_33_15 PM.jpg

File synchronization: moving away from incremental backup (HELP!)
Files to Protect Screenshot

Screenshot - 11_29_2007 , 6_33_07 PM.jpg

File synchronization: moving away from incremental backup (HELP!)
Email Protection Screenshot

Screenshot - 11_29_2007 , 6_32_58 PM.jpg

File synchronization: moving away from incremental backup (HELP!)
Remote Storage Screenshot

Screenshot - 11_29_2007 , 6_21_56 PM.jpg

File synchronization: moving away from incremental backup (HELP!)
Advanced Settings Screenshot

Screenshot - 11_29_2007 , 6_32_49 PM.jpg

File synchronization: moving away from incremental backup (HELP!)

f0dder · « **Reply #31 on:** November 30, 2007, 06:32 AM »

Thanks, John...

Offers to only send the portion of the file that has changed vs. entire file again

Do you have any idea if it does this with the help of a filter driver, or (like SFFS/whatever) by scanning the file for changes?

johnfdeluca · « **Reply #32 on:** November 30, 2007, 08:51 AM »

Do you have any idea if it does this with the help of a filter driver, or (like SFFS/whatever) by scanning the file for changes?
-f0dder (November 30, 2007, 06:32 AM)

I don't know but here is the relevant portion of the Help File:

Use sub-file copy check box
Set sub-file copy for remote backup copies.

Initially, an entire file is copied to the storage areas. When sub-file copy is turned on, and when the file changes, only the changed information is copied to the storage area. The sub-file copies are saved as separate files on the remote storage.

Sub-file copy can significantly reduce the amount of network traffic. However, sub-file copy consumes more processing resource on your computer. The default setting is to use sub-file copy for files larger than 50 MB. If you need to conserve more network resources, you can reduce the size setting so sub-file copy will be used on even smaller files.

Check the box to turn on sub-file copy. In the Use sub-file copy for files larger than: field, specify the file size threshold for using sub-file copy. For files larger than this size, only the changed information is copied to the storage area.

And from the Redbook:

Sub-file copy and versioning
If you configure Tivoli Continuous Data Protection for Files to use sub-file copy, it will create files that contain each file's delta information in your designated storage area. Those files have an FPdelta file name suffix.

To test the sub-file copy functionality the file Sub-file example.txt was modified. This file is larger than 3 MB. Figure 4-23 shows the replicated directory in our HTTP Server Paris. As you can see, we have the original file with 3,208 KB and a delta file with only the modified characters which is only 1 KB. Note that this feature is supported only with remote storage.

To track versions of a file, Tivoli Continuous Data Protection for Files adds a version suffix to the file name of the backup copy. On remote storage, all backup copies contain a version suffix. When a file is deleted on your computer, Tivoli Continuous Data Protection for Files will add a version identifier to the file name of the most recent backup copy on the remote storage.

The most recent backup copy of a file is the active backup copy. Older backup copies of that file are inactive backup copies. Tivoli Continuous Data Protection for Files will delete inactive backup copies of a file first if the storage space limit is reached.

A file that is protected by schedule could change several times during the schedule interval. Only the last version of the file prior to the end of the schedule will be backed up. A continuously protected file is backed up after every change.

And finally, from the Tivoli Storage Manager Redbook:

Adaptive subfile backup
Traditional storage management techniques have focused on protecting systems that stay in the same place and are always connected to the network. Today, many workers are mobile and/or remote, and keep critical data assets on their mobile computers (laptops) and other wireless devices. Storage administrators need new ways to protect mobile and remote computers with limited access to the infrastructure that serves the rest of the company. Some limitations include being attached to the corporate network with reduced bandwidth, limited connect time, and minimal assistance to perform the backup.

This limited access both increases the criticality of storage management services and limits the applicability of traditional methods and policies. Tivoli Storage Manager helps resolve these problems with its adaptive subfile backup feature, which reduces the amount of data transferred while backing up changed files.

This features enables the backup-archive client to back up only the changed portion of a file, either on byte level or on block level, instead of transferring the whole file to the server every time. The changed file portion is backed up as a differential backup relative to the last complete backup of the file (base or reference file) and it is called delta file. All changes since the last complete backup of the file are included in this delta file. In the case of a restore, this allows for the restore of the whole file by restoring only two sub-file components, one delta file, and the last complete backup of the whole file, the base file.

The decision to base the differential on byte level or block level will be made at the backup of the base file, and it depends on the size of the file. Subfile backup technology is not used for very small files (less than 1 KB in size) or for very large files (bigger than 2GB). If the delta file size exceeds 60 percent of the base file at the last sub-file backup, a new base file will be transferred.

The adaptive subfile backup, as well as the restore of a file consisting of a base file and the delta file, is completely transparent to you as a user. All necessary file data separations or reconstructions happen under-the-covers of the backup-archive client. Also, all other Tivoli Storage Manager features, such as policy management or fault-tolerant backup and restore, still fully apply. Adaptive subfile backup is used for incremental as well as for selective backup. It is aware of multithreading and will work together with client data compression and encryption.

nitrix-ud · « **Reply #33 on:** November 30, 2007, 10:51 AM »

just my 0.05 cent
and please bear with me if it's already been said

by far, the best solution i've seen for folder sync is FolderShare (http://www.foldershare.com)

there are however some problems (which can be deal breakers !)

10000 file limit per folder (you can sync up to 10 folder)
sometimes the folder share server does not work, and in that case the sync does not work...

for my backup need i use SyncbackSE (which is very good)

well in any case FolderShare is the best app i know along with autohotkey and FARR

so you may give it a try, if the limitations are ok with you
i use it for collaborative work and it's just such a productivity boost...
i could not live without it
FolderShare is the only app i know that is really unobtrusive and real-time...
from what i recall mirror folder is real-time until the destination is offline... (am i wrong here ?)

I've also tried (but not fully tested) PowerFolder (similar to FolderShare)

Cheers, Nitrix

johnfdeluca · « **Reply #34 on:** November 30, 2007, 11:11 AM »

10000 file limit per folder (you can sync up to 10 folder)
-nitrix-ud (November 30, 2007, 10:51 AM)

10 folder limit seems pretty limiting.....Is this a free version limit?

tranglos · « **Reply #35 on:** November 30, 2007, 11:46 AM »

10000 file limit per folder (you can sync up to 10 folder)
-nitrix-ud (November 30, 2007, 10:51 AM)
10 folder limit seems pretty limiting.....Is this a free version limit?
-johnfdeluca (November 30, 2007, 11:11 AM)

Looking at the website, it's an online storage thing, ie. your data kept on their servers. It does not appear to sync folder to folder, locally:
https://www.foldersh...mages/screenShot.jpg
https://www.foldersh...info/howItWorks.php?

John, how's Tivoli Continuous Data Protection doing? I've just installed the trial on my laptop and it's very interesting, but I haven't quite figured out its workings yet. I've noticed it starts backing up certain files even without any manual configuration, which is nice. Nicer even that it picked up my Firefox profile folder of its own volition. At the same time there's a weirdness to it, because from the whole profile folder it picked and backed up only one XUL file. (I know very little about how it works yet, though I acquired some familiarity with Tivoli Storage Manager, when it was being localized for Polish some years ago. I remember TSM had a very cool idea of archiving with stub files, but CDP does not seem to include that.)

I've been putting off installation of the trial since IBM customer support cannot figure out if I can even buy the product from them. Poland is not listed among the countries on the product page, and when I try to go to checkout, I get an "unexpected error" from the server. IBM support keeps redirecting me from person to person, and it doesn't look like I'll be able to order it, so I'm hesitant to use the trial, since it looks like a GREAT solution for hands-free backup to NAS, and I don't want to get attached

.marek

johnfdeluca · « **Reply #36 on:** November 30, 2007, 01:20 PM »

Sorry to hear of your problems navigating the IBM maze....wish I could help. As far as my experience with it, it's been all good. That said, after installation a year or so ago, I've not really spent much if any time in the UI until/except to help provide info here. That's one of the things I like about it....set it and forget it. It selects files for synch which meet the criteria I set which is either a filename/ext I set or via the programs list (which seems to know what data files to back up). But my experience is that it doesn't back up anything under it's own volition....it has to meet the selection criteria I specified in the settings.

Please post back your experience with IBM in resolving this problem.....I think (good or bad), it's important folks know about it, especially if they are considering a purchase. While the backing of a big company like IBM behind software can be a good thing in terms of knowing the company will likely be there to support the software....it can also be a bad thing if the little guy gets lost in the shuffle of impersonal treatment.

nitrix-ud · « **Reply #37 on:** November 30, 2007, 01:31 PM »

10 folder limit seems pretty limiting.....Is this a free version limit?

nope this is the limit, there was a paid version before microsoft bought it... (maybe without the limit... i don't know)
i would pay a lot to remove it though...

Looking at the website, it's an online storage thing, ie. your data kept on their servers. It does not appear to sync folder to folder, locally:
-tranglos

it has nothing to do with online storage

(no file is stored on their servers...)
and it DOES sync folder to folder either locally or through the internet

(it does need the foldershare servers to manage the sync process however)

tranglos · « **Reply #38 on:** November 30, 2007, 01:41 PM »

it has nothing to do with online storage (no file is stored on their servers...)
and it DOES sync folder to folder either locally or through the internet
(it does need the foldershare servers to manage the sync process however)
-nitrix-ud (November 30, 2007, 01:31 PM)

I apologize, I must heve gotten confused by their screnshot, then

But what about the "web download" feature, which apparently lets you get your files from any computer, anywhere, by connecting to the foldershare server?
https://www.foldersh...help/learnWebDL.php?

On edit: I think I got it. It seems to create a p2p connection, so as long as the source machine is connected to foldershare, you can connect any other machine anywhere and download files from the source.

.marek

Armando · « **Reply #39 on:** September 28, 2008, 10:16 AM »

Sorry to revive this old thread.... But I couldn't find the info I'm looking for and I'm curious about what f0dder chose -- in terms of backup/mirroring -- for the museum he works for.

So, f0dder : did you stick with MirrorFolder after all, and does it work well (in terms of performance, HD access especially...)? I'm pretty happy with SyncBackSE, but MirrorFolder seems so cool.

MrCrispy · « **Reply #40 on:** September 30, 2008, 01:35 PM »

I can't tell for sure from the MirrorFolder site, but it seems that it does not support bidirectional sync. Is this correct? This is a very important feature as the primary use for me would be to keep my files synchronized across all of my pc's, andhaving them in sync automatically means they are backed up in multiple locations as well.

f0dder · « **Reply #41 on:** September 30, 2008, 01:42 PM »

Not sure if it does - and I don't think it's the point of the application anyway (I can see how it would fit with the name, though $:-\$ ).The filter-driver mirroring method is darn useful though, and I haven't seen it in other apps...

4wd · « **Reply #42 on:** September 30, 2008, 02:35 PM »

I can't tell for sure from the MirrorFolder site, but it seems that it does not support bidirectional sync. Is this correct? This is a very important feature as the primary use for me would be to keep my files synchronized across all of my pc's, andhaving them in sync automatically means they are backed up in multiple locations as well.
-MrCrispy (September 30, 2008, 01:35 PM)

From their website:

Synchronization between laptop and desktop computers - You can use MirrorFolder to synchronize files between your laptop and desktop computers using bi-directional automatic synchronization mode.

f0dder · « **Reply #43 on:** September 30, 2008, 02:52 PM »

4wd: I was thinking specifically about the mirror-driver automatic syncing mode, I can see a lot of potential for that behaving... interestingly... on bi-directional sync

MrCrispy · « **Reply #44 on:** October 01, 2008, 05:38 PM »

I'm going to have to try this out. I like SFFS because it also handles file moves and is very configurable (maybe too much). If MirrorFolder can match its sync abilities and is faster then I might switch.

tranglos · « **Reply #45 on:** October 01, 2008, 05:56 PM »

If MirrorFolder can match its sync abilities and is faster then I might switch.
-MrCrispy (October 01, 2008, 05:38 PM)

In the filter driver mode MirrorFolder is instantaneous. This one feature is fantastic, but there were too many things I didn't like about MF. I finally chose FileHamster for real-tome backup, even though I don't like .Net apps in general. From my notes, these are the issues I had with MirrorFolder:

No option to enforce a minimal delay between backups
No option to archive files without zipping them. This makes file comparison awkward.
When archiving, each modified file is saved to a separate zip file, which is not named after the file, but has a name like "zip_YYYYMMDD.zip"; so finding, comparing and restoring archives is awfully inconvenient.
Very hard to restore, since one has to manually select individual files, and the dialog boxes don't even open in the defined backup location. Quite inconvenient in comparison to FileHamster. Hard (or impossible) to see which files have how many archived versions.
No option to specify the number of archived versions to keep (there IS an option to delete archived versions older than n days though)
Never received a reply to a support question I asked during the trial period

urlwolf · « **Reply #46 on:** October 02, 2008, 12:52 AM »

Tranglos,
I think you are trying to use MF to do versioned backups (at least that's what fileHamster does). And my guess is that MF is designed for synch'ing not for versioned backups (although most of these programs can be used for both see e.g., the very name of synchBack).

MerleOne · « **Reply #47 on:** October 02, 2008, 01:22 AM »

This thread reminds me of an excellent real-time backup software called AJC Backup where versionning is very nicely done. Versions are saved in a proprietary compressed (or not) format which is very efficient at storing multiple versions of a same file. I may have 20 versions of an .xls file and the archive is 70% of the size of the last xls file.

[attachthumb=#1][/attachthumb]

Backup is updated as soon as a file is saved.

More info on http://www.ajcsoft.com/AJCActBkTour.php

f0dder · « **Reply #48 on:** October 02, 2008, 02:28 AM »

MerleOne: how does it do it's job? Filter Driver, or filesystem notifications?

MerleOne · « **Reply #49 on:** October 02, 2008, 03:43 AM »

MerleOne: how does it do it's job? Filter Driver, or filesystem notifications?
-f0dder (October 02, 2008, 02:28 AM)

I would bet on filesystem notification, since there is no special service/driver running AFAIK, but the developer should probably be asked.