Author Topic: How do incremental backup programs work? (Read 24,592 times)

jgpaiva · « **on:** June 17, 2008, 10:47 AM »

Ok, so as i know most people here know more about backup than i do, I'm interested in asking you 2 things:
If you change one bit in a 8GB file, how much space does the incremental backup take? does it add just a few kb, or the whole 8GB? More specifically, does the incremental backup work by whole files or byte by byte?
How about if you change the name of an 8GB file? Does it consider that as a new file, or is it smart enough to tell it's the same file with a new name?

Sorry, just another question: I know some backup programs (that do incremental backup) allow you to see the files you've backed up. How does that work? Do they come up on Windows Explorer (as a virtual drive or something) or do they use a sort of "Backup Explorer", a proprietary application that works similarly to WE but can only be used for backups?

Thanks a lot!

justice · « **Reply #1 on:** June 17, 2008, 11:09 AM »

AFAIK,
Incremental Backup will back up the complete file if the file has changed, so 8GB. But if you have 1000 files which total 8GB then only those that have changed are backed up.
Block Level Incremental is " A more sophisticated method of backing up changes to files is to only back up the blocks within the file that changed. This requires a higher level of integration between the filesystem and the backup software." which would only back up a few KB.

Your other questions depend on the backup software that you're looking at, for example SyncToy (not a backup program really) and SFFS keep a database of changes so that it knows you renamed your 8GB file and can do the same on the other end without uploading anything, most other programs do not. Also some programs such as Jungle Disk that I wrote about create a network mapped drive to make their files available, others don't. This usally depends on whether or not the final backup is filebased or image based (ie, the whole backup is one or more split archives or whether they are folder structures in the file system). Hope that helps

tomos · « **Reply #2 on:** June 17, 2008, 11:17 AM »

AFAIK incremental backup usually means they will back up the files that have changed (again) and ignore the files that havent
That usually means full file is backed up

Then there's partial file update (probably known under different names) which is used with synching -
the changed "bit" in your original 8GB file gets updated in the synched file
Obvious problem here being if your file corrupts you have no backup as such
SFFS does this
SFFS keeps track of files if moved (says it does anyway) but not sure what happens if the name is changed...

thats about my limit [Justice got in ahead of me there...]

mouser · « **Reply #3 on:** June 17, 2008, 11:17 AM »

Quote

Sorry, just another question: I know some backup programs (that do incremental backup) allow you to see the files you've backed up. How does that work? Do they come up on Windows Explorer (as a virtual drive or something) or do they use a sort of "Backup Explorer", a proprietary application that works similarly to WE but can only be used for backups?

Some programs mount the backup file as a virtual drive that shows up in explorer (Acronis for example) and some use their own backup explorer (Genie backup for example). And still other simply use a standard archive file format for backup so you can open them in your own tool.

Perry Mowbray · « **Reply #4 on:** June 17, 2008, 11:19 AM »

I'll answer this one...

Quote from: jgpaiva on June 17, 2008, 10:47 AM

Sorry, just another question: I know some backup programs (that do incremental backup) allow you to see the files you've backed up. How does that work? Do they come up on Windows Explorer (as a virtual drive or something) or do they use a sort of "Backup Explorer", a proprietary application that works similarly to WE but can only be used for backups?

It depends on how the actual software stores the backup.

For example, Backup4All gives you the option of zipping the files or not. You can browse the backups using the file system, though if you are keeping versions it can get a little confusing (better to use B4A). From memory SyncBack is similar.

FirstDefense though stores it's data in an archive file of some description that I've never tried to access through the filesystem.

Just to reinforce Justice's explanation:
Incremental refers to the Backup as a whole, not the individual files being backed up.

Quote from: WikiPedia

An incremental backup is a backup method where multiple backups are kept (not just the last one). These backups will be incremental if each original piece of backed up information is stored only once, and then successive backups only contain the information that changed since a previous backup.

jgpaiva · « **Reply #5 on:** June 17, 2008, 11:21 AM »

I probably should have been more specific.
Well, the thing is: i'm now entering the last year of my masters. That means I'm going to be the whole year working on my masters thesis.
The theme will be "Backup Chunk - a backup system based on chunks". This corresponds to the "Block Level Incremental" that you mentioned, justice. What i meant with my first post was to know if any of the most known programs use the Block Level Incremental, as the people around here know more about these systems then i do

I've already looked at SyncBackSE, and it doesn't have the "block level incremental", but i don't know about the second part (is there a "backup browser" or not?).

jgpaiva · « **Reply #6 on:** June 17, 2008, 11:25 AM »

You guys are incredible.. 3 replies while i was posting mine above

So, the general opinion would be that the best programs (that do have incremental) mount the files as a virtual disk, and don't have block level incremental?

tomos · « **Reply #7 on:** June 17, 2008, 11:28 AM »

Quote from: jgpaiva on June 17, 2008, 11:25 AM

So, the general opinion would be that the best programs (that do have incremental) mount the files as a virtual disk, and don't have block level incremental?

well, if a programme exists that does "block level incremental" backups I'd like to know about it & would most probably WANT it as well

mouser · « **Reply #8 on:** June 17, 2008, 11:31 AM »

Whatever software you go with, make sure that you can recover multiple previous versions of older files -- not just the last backup.

And I recommend multiple backup solutions -- i like to use BOTH an incremental backup solution AND an occasional drive imaging. If one fails, i have the other to fall back on. And make sure you keep occasional backups online or outside of your house in case of a home disaster.

justice · « **Reply #9 on:** June 17, 2008, 11:33 AM »

Jungle Disk + JD Plus does block level incremental backup

http://www.jungledisk.com/plus.shtml for more info.

mouser · « **Reply #10 on:** June 17, 2008, 11:36 AM »

Block level incremental is not something i look for.

It's not hard to think up scenarios where block level incremental could save you backup space -- for example if you have a lot of LARGE files for which you were only adding stuff at the end of them (giant log files). But that's not that common, and i don't have any such documents of my own that i would mark for incremental backup.

For me, block-level incremental backup seems to prone to trouble, and i feel better having full copies of each version of each backed up file. I don't want all the backups to be dependent on one another the way they would be with a block-level thing. That's just asking for trouble in my view.

As far as the "best" programs supporting virtual mounting of backups -- I wouldn't say that. However coding such a feature is pretty advanced stuff, so when you see that feature it's probably a pretty good indication that you are dealing with an advanced program. It can be quite a useful feature sometimes if you have your own file search/explore tools that you would like to use on your backups.

ps. it's hard to post to this thread because each time you try to post you get "Warning - while you were typing a new reply has been posted. You may wish to review your post."

jgpaiva · « **Reply #11 on:** June 17, 2008, 11:39 AM »

jgpaiva drops an anvil on justice! Bah, that's bad news for me

mouser: the good advantage of using a block level incremental backup system is that to store multiple versions of a file, you only have to store the diffs and the metadata to be able to rebuild the file.

jgpaiva hopes JD doesn't get well known

[edit] oh, JD needs the S3 subscription.. I feel better now [/edit]

jgpaiva · « **Reply #12 on:** June 17, 2008, 11:47 AM »

@mouser: yeah, block-level might not seem useful for the regular user, but there's one more advantage: similarities between files also are recognized, thus, the compression is made throughout the whole backup and not only per-backup. I probably didn't express myself correctly.
What i mean is that when you do a backup, it is compressed. But it is compressed only based on information from the current backup, thus, it takes more space then it would if it used information about previous backups (which is possible with block-level backup).
I'm not absolutelly sure about this, but i think that conceptually it is possible to take as little space as possible, by using BLB.

The point you made about the whole backups depending on each other is interesting, though... I think i'll have to think about a solution for that

justice · « **Reply #13 on:** June 17, 2008, 11:50 AM »

Maybe you can look at how PAR files work which means you could use small helper files to reconstruct the backup even when the whole backup wouldn't be available? I always thought they'd be very useful even in a downloadmanager.

Parchive parity files (or par files for short) create redundant data that can be used later in case parts of the original data is lost or corrupted. Par files allow file level recovery of data. That is, out of a group of many files, if a limited number of files are lost or corrupted, they can be recovered. These files will have the extensions of .par, .p01, .p02 etc http://en.wikipedia.org/wiki/Parchive

jgpaiva · « **Reply #14 on:** June 17, 2008, 11:51 AM »

Quote from: justice on June 17, 2008, 11:50 AM

Maybe you can look at how PAR files work which means you could use small helper files to reconstruct the backup even when the whole backup wouldn't be available?

Yep, that was my first idea too

It'll take some more investigating to know if it'll actually compensate the space it takes, though

Chris · « **Reply #15 on:** July 12, 2008, 07:16 AM »

One of the best known (in the Linux world) incremental backup program is rysnc http://rsync.samba.org/. It is used extensively to synchronize mirrors of files etc . It is explained at http://rsync.samba.org/how-rsync-works.html.

I use it on a Windows PC to backup to a Linux server. It is a command line program, not a GUI but when setup works very effectively.

Chris

jgpaiva · « **Reply #16 on:** July 12, 2008, 08:57 AM »

Rsynch looks very good!
Thanks a lot for the pointer, Chris. I'll give it a good read

jgpaiva · « **Reply #17 on:** September 25, 2008, 08:52 AM »

Hi!
Sorry for digging up this thread, but I wanted to ask you guys something:

Does your backup system allow you to save multiple backups to hdd, and access them on a snapshot basis? Like.. "how was my system 30 days ago?" Or do you have to check the incremental backup made at the time and if some file isn't there check the previous one and so on? Or does it tell you: "this file is on backup X, please insert CD"?

Thanks!

Armando · « **Reply #18 on:** September 28, 2008, 06:33 AM »

I use a not exactly orthodox incremental backup method with SyncBackSE. All new files or files that have changed are backedup everyday/several times a day, and compressed in a zip file. I keep all these compressed files, and when my external HD gets rather full, I dump them on DVDs and archive the DVDs (date, etc.). But this has to be done only once every couple months maybe, or more. Then, when I'm looking for something, I use 2 different methods to find specific files : WinRAR's searching capabilities inside encrypted multiple compressed files (works pretty quick and well), or Archivarius. I used to use Archivarius (each time I put stuff on a DVD, I indexed all zipped files content in Archivarius --> excellent!) but I stopped doing it when I decided that everything would be encrypted... Of course, if I was using TrueCrypt, it would still be possible... But I haven't managed to devise a new strategy including truecrypt.

f0dder · « **Reply #19 on:** September 28, 2008, 02:31 PM »

Block-level backups... that would have to either be closely integrated with the filesystem (could be done as a filter driver in Windows - see MirrorFolder, their (badly named) "RAID-1 mirroring" option does this (oh, and crosslink to this thread seems appropriate)). If you don't go for this kind of integration, you'd have to compare source/dest files to only backup changes, this could be very costly.

On linux I use rsnapshot, which is pretty nice - it has "last X {hours, days, weeks, months, years}" like the Apple TimeMachine, implemented through use of hardlinks (ie., minimal disk usage - doesn't have changed-blocks though, so a bit change in an 8gig file will still mean a new 8gig file). It's not super suited for backing up my workstation, though, since the backup machine "pulls from the client", instead of the "client pushing to the server".

jgpaiva · « **Reply #20 on:** September 28, 2008, 02:38 PM »

F0dder: thanks for the pointer to filter driver! I think that's pretty much what I'm looking for to monitor the changes to files, right?

f0dder · « **Reply #21 on:** September 28, 2008, 02:46 PM »

Quote from: jgpaiva on September 28, 2008, 02:38 PM

F0dder: thanks for the pointer to filter driver! I think that's pretty much what I'm looking for to monitor the changes to files, right?

Yup.

You could use a usermode filesystem change notification instead, but that only tells you "there's been changes in this folder" (or subfolder tree), then you need to scan the folder (or entire subfolder tree) for changes - and then you'd have to scan those files for updates.

So filter driver is definitely the way to go, since you get notified of each individual write.