Bvckup

[apankrat]

Ahem... tap tap tap... Is this thing on?

Hi,

I would like to introduce a little creation of mine called Bvckup.

In short, it is a folder cloning utility. One points it at a pair of folders and it makes sure the second folder is exactly like the first one.
Do I see a puzzled expression on your face? Not a rocket science, you say? Just give me a minute :)

---

First
Bvckup's default mode of operation is to constantly monitor for changes. Once the changes are detected, it propagates them to the destination. This makes Bvckup a real-time backup utility.

Second
Its default copying mode is delta copying. Which is a fancy way of saying that it copies only changed parts of the file. This makes Bvckup fast, really fast. Lots more details on why and how along, mind you, with the list of drawbacks of this particular approach is available here - Delta copying, explained.

Third
Bvckup is not a two-way sync utility, it's a backup utility and it makes full use of the core difference. By default it assumes that the destination copy does not change between the backups. This allows it to not scan the destination directory to detect what changed, but rather rely on a locally saved summary from the last backup.

I hear you say "bah, so what?". Two words - NAS backups. Not needing to scan the destination directory eliminates a lot of network traffic, and it removes the need for spinning up otherwise idle disks on NAS devices. This adds up to some major time savings, on top of those delivered by delta copying.

---

Still with me? A bit more perhaps? :)

Fourth
The user interface. It is not about utilizing all 16 million colors in a single toolbar icon. Verbosity is not a feature, multitude of options is not a convenience. Properly designed user interface is all about simplicity and unobtrusiveness of the day-to-day interaction flow. A small example would probably go a long way here - Bvckup toolbar - but the only sure way to appreciate the UI polish is to take the app for a spin.

In fact Bvckup is as much about the UI design as it is about technical features. This bit explains how tightly these two are intertwined in case of Bvckup and how they shaped the design of the app - Short history of Bvckup.

Fifth
It is small and very lean. The installation package is about 480 KB. That's as much as a splash screen logo of some other backup products :) The app is written in C++ but without any extensive use of ++ features. It's much more like C with Classes... just like Bjarne envisioned before he got sucked into that bloody language design committee... but I digress.

Sixth
It is technically sophisticated. It doesn't copy just the file contents and the attributes, but also the timestamps, NTFS security and ownership information. If it finds a file that it cannot copy (locked, open, etc), it will try and use Volume Shadow Copying to work around the problem. If it is running under stripped-down administrator account on Vista or W7, it will prompt and offer to elevate itself to full admin privileges.

Also, and it's for true Windows connoisseurs only - it can run in elevated and filtered modes using the same executable. In fact, the even more amazing feat is that the very same executable can interact with Shadow Copying service on both XP and Vista; even though Microsoft explicitly requires linking to two different VSS libraries for these two platforms. Took a while to figure this one out, I can tell you that :)

---

A bit more...
Assorted bits and pieces of some interest - built-in software updates, linear regression ETA estimator, the built-in interactive Mini Guide shown on the first run, a bit of a logo and web design goodness and a bit about myself.

And lastly...
All beta versions are free, with some tasty perks in store for the beta users.

---

So that's about it. I hope you got the feel for what this program is about, and what the design approach behind it is.

Here is the URL - http://bvckup.com - check it out, let me know what you think.

Thanks,
Alex

Screenshot:

[sajman99]

apankrat, congrats on your software. I've been using Bvckup for a short time and am quite content with it.

I think many more folks will be using Bvckup as it becomes more well-known. It surely deserves recognition as the excellent program it is.

Good luck to you.

[cranioscopical]

Interesting stuff, Alex.
Keep it up!

[Armando]

Yes, very interesting.

I'm currently testing bvckup's latest beta.

1- I'm syncing a 215 GB partition to an external HD, USB 2, not the fastest let me tell you. The first Sync took a while, but that was completely expected. Files first needed to be copied fully.

2- RAM usage started at around 15 MB, peaked at 90 MB during syncing. CPU usage was more than acceptable. I could still do stuff while the syncing was taking place. No errors whatsoever, shadow copy is ON, which means that every single file got copied, even the "in use" ones.

3- subsequent syncings :

a- RAM usage stays at around 60 MB afterwards. Similar to other apps in that category -- more than syncback, much less than Oops!Backup.

b- fast as hell. Less than 7 min each time, about 4 min for the last one.  Here's what the log file says after the last sync (only parts of the modified files are being copied, thanks to the delta copying technology) :

2010-10-14  22:16:14  Job [SyncAllDocXimeta] completed in 4 minutes 11 seconds with 0 errors, copied 228 KB out of 1221 MB in 49 files

Just as you guys know... 228 KB out of 1221 MB (1 250 304). That's about 0.018 % of the total size. Isn't that great ? Instead of copying the whole 1221 MB like SyncBack does, only 228 KB were copied (i.e : the modified portion of the files.)

I don't backup to any network drive, but if I was, it'd be good news.

I need to do more testing of course.... have to somewhat moderate my enthusiasm... but it seems that this could finally replace Syncback or SyncToy as a syncing solution. Especially if apankrat implements versioning and exclusion of folders for the scanning part (when folders are actually excluded from copying)... Oh, and proxy support.  

 

[Armando]

I had a look at your web site... Impressive. (I like how it looks like too... and I also like bvckup's web site AND its UI. Very slick, simple, to the point.)

So you're also the coder behind Hamachi ? (Congratulations, really, if that's the case...). The way LogMeIn uses it certainly proves its efficiency.

[apankrat]

@sajman99 - thanks, great to hear you like it :)
@cranioscopical - thanks

@Armando -

One thing to keep in mind with regards to the copying speed is the following. With delta copying there's always a small chance that a change will be missed because a block of data happens to have the exact same checksum (or hash) after it is modified. Bvckup is using MD5 over moderately sized blocks (32 KB), so technically the probability of such event are really really small. However, just to err on the side of caution, Bvckup will copy every file in its entirety every N backups (8 by default). This is configurable in backup Properties, Backup What section, behind the Details button.

Glad you like the website. My hobby du jour - http://dribbble.com/players/apankrat :)

Regarding Hamachi - yeah, that was my baby too. It was wildly popular in gamers' circles, but I doubt it still is as LogMeIn repositioned it as an IT/sysadmin tool.

[Deozaan]

I've been using SpiderOak to synchronize game save files between computers, and recently had the brilliant idea of using it to sync my Minecraft saves. Unfortunately SpiderOak uses up way too much CPU and slows my system (and Minecraft) to a crawl because it syncs while in the game.

Well, obviously I can't use bvckup to sync between computers, but I did just test it out backing up Minecraft saves/worlds and am happy to report that it didn't cause the game to run sluggish, and it didn't seem to affect my system speed at all.

A bug I noticed: In the log file, if I scroll to the right to see the end of a log message, all the text will magically jump to the right, but the scrollbar will still be scrolled all the way to the right, too, which keeps me from being able to read what I had scrolled to read. I don't know if it's a scrollbar error or a text display error, but something is broken.

Re: Hamachi - I still use it to play games (e.g. Diablo II) with family and friends across the internet. Thanks for that incredibly useful tool!

[MerleOne]

@Armando - not to mention it was made in Canada
@mouser - done - https://www.donation...ex.php?topic=24261.0
@MerleOne - have a look here, let me know what you think - http://www.bvckup.co...opic_show.pl?pid=258

-apankrat (October 14, 2010, 08:38 PM)

Thanks for making the link with my other post !  I guess I missed you answer on your forum.  I think my preference would go to a special folder containing previous revisions, some kind of archive folder.  But it would be great if you could use, as an option, winrar solid archive to store revisions of a same file.  Solid archiving (in winrar) is then deadly space-efficient (while taking some more time).  What I am concerned about is the space taken. 

For instance, when I edit a big excel file, I save it each time I make a critical change, this can be 10-20 times per day, sometimes even more.  If the file is 4 MB big, without a clever way to handle the versions, it would take a huge space on my HDD.

Also, the next step would also be to manage those revisions, purge them at times; either manually or with a specific setting.

[MerleOne]

BTW, I am forgetting something important : while performing some tests a few weeks ago, source folder was on a local HDD and destination on a network folder, I encountered strange issues with previous build, that garbled the ownership permissions on the destination, such that I couldn't even open/read backup-ed files.  I managed to repair some attributes.  I usually have no problem whatsoever with that network drive.

I remember it all started with an error message from bvckup looking like "can't set ownership on dest files".  I'll give it another try next week with the new build.

I didn't mention it right away because I was busy and it's very difficult to explain in details what happens.  But there was definitely something wrong when using  my network drive, which usually shows no issue regarding file attributes.

[MerleOne]

Thanks to have moved my previous 2 messages to this thread.   

[jaden]

I already had the site bookmarked but only vaguely remember reading about it.  I never tried it out, but that has now been remedied.  Great work!

With Foldershare/Live Sync going away and the replacement not working on XP, are there any plans to make this work across remote machines?  This simple interface with remote support would be killer.  Any plans for a portable (non-installer) version?

The web site looks great too.  I followed the Yaletown Software Design link to read more about the design team and was surprised to see it was you.  Being a skilled developer as well as a talented designer is a rare, but fantastic combination.  The UI design of Bvckup is a great example of what that combo produces.

Great work on Hamachi too - I hope you were well compensated by LogMeIn

[Perry Mowbray]

Any plans for a portable (non-installer) version?

-jaden (October 15, 2010, 03:25 AM)

The current beta is portable... I installed it on my USB at my (very restrictive) work today without problems 

[MerleOne]

Another remark on the filter panel :
1/ Is a '\' mandatory at the end of "location" field ?
2/ Would it be possible to edit and not jut remove the defined exclusions ?

[jaden]

The current beta is portable... I installed it on my USB at my (very restrictive) work today without problems

Cool, thanks for the info.

I'd still prefer a zip instead of an installer so it wouldn't need to be installed at all, but at least it works from a USB drive.

[Nod5]

Impressive features! I'm using SyncBack but the deltacopy feature sounds useful so I'll test switching to this.

I only have one questions: how do you pronounce the name?  "bvackup"? "bee-vee-cup"?

[Armando]

I did some more tests.

Bvckp is usually faster than SyncBack here, but not that much, depending on the types of files copied.
Sometimes it can even be slower. But I haven't tested anything over a network.

apankrat : it seems that maybe bvckup doesn't detect file renaming / moved files so that it can perform the same operations on the destination, instead of deleting / copying the files anew. SyncToy does that and this means dramatic speed improvement.

george7987 makes some interesting comparisons there


Created a backup job which basically syncs a bunch of big files. I made SyncBack profile so that it's the fastest possible

[EDIT : Need to redo the test to make sure numbers are correct... Sorry  :-[]

As far as performance goes for these operations, there's room for improvement when
- "raw" copying from source to destination
- file and folder renaming in the source

Of course :

- SyncBack or SynToy don't do delta, Bvckup does...
- SyncToy doesn't use VSS, Bvckup does.

[EDIT : note that I'm trying to be constructive here... Bvckup is still beta and free at the moment.  ]

Edit : Sync Back seems to take care of renaming (both files and folders), File move and , after all... Only for some reasons, it takes longer after the rename for it to take it into account. No idea why. Back in a few minutes... Nope it doesn't 

[Armando]

Posted the results of my tests there

Thanks.

[KynloStephen66515]

@sajman99 - thanks, great to hear you like it
@cranioscopical - thanks

@Armando -

One thing to keep in mind with regards to the copying speed is the following. With delta copying there's always a small chance that a change will be missed because a block of data happens to have the exact same checksum (or hash) after it is modified. Bvckup is using MD5 over moderately sized blocks (32 KB), so technically the probability of such event are really really small. However, just to err on the side of caution, Bvckup will copy every file in its entirety every N backups (8 by default). This is configurable in backup Properties, Backup What section, behind the Details button.

Glad you like the website. My hobby du jour - http://dribbble.com/players/apankrat

Regarding Hamachi - yeah, that was my baby too. It was wildly popular in gamers' circles, but I doubt it still is as LogMeIn repositioned it as an IT/sysadmin tool.

-apankrat (October 15, 2010, 12:56 AM)

Still very much in use in the world of gaming, especially for private alpha testing of new games and/or private servers.

I can tell you that is is very much used for use on "Conquer Online 2.0 Private {Pirate} Servers"

Im a user of Hamachi and am very very impressed with it (In fact, it is currently on my taskbar, fully loaded and connected to 6 networks (4 of my own).

I also use this as a system admin tool and have introduced it to a collegue of mine for use within the office as a Lan Chat and File sharing utility with the obvious option to ping computers to check quickly if they are switched on or not.

I may write up a review of this actually, seeing as I am a regular user and rather like it, and much prefare it to the other options available.

Love Hamachi, and will certainly be trying this one out over the weekend.

[apankrat]

@Deozaan - the scrolling bug - I can reproduce it, but it appears to be a quirk of a standard RichEdit control. Should be easy to fix, thanks for the report. Hamachi + Diabolo --- lol, yeah, that was one hell of a combo :)

----

@MerleOne - with regards to backing up into an archive file - I am very reluctant to add this. Another person asked for a similar feature over an email, let me just copy-paste my reply here:

It is indeed a nice feature, and you are not the first one to ask for it. I can see how it can be useful and how it is related to what the program does at the moment. However it complicates the program. Not in terms of complexity of the code, or even the complexity of the UI. It's all about making the software that does just one thing and does it really well. Right now it is very easy to explain what the program does - it makes a copy of a folder. Once advanced options are added, like the one you are suggested, the program starts to turn into a Swiss Army Knife. Its scope blurs, it becomes harder to understand what it is for exactly. It sure becomes more featureful and could potentially accommodate the needs of a wider variety of users, but at the same time it will become less and less appealing to those who just need to just copy a folder. In the end by adding a feature the program may gain less users than it would loose - loosing those who view major unwanted features as a bloat or unneeded complexity.
 
One option would be to make another, separate program that would backup into the compressed archived. And do only that.

With regards to the "can't set ownership on dest files" issue - I saw this before, and it was triggered by a file being initially copied from another machine or created by another user. So bvckup ended up making a copy of the file not owned by the user whose account it was running under. The first copy would succeed, and the second copy will fail, because the destination machine will (quite reasonably) refuse bvckup full access to the file owner by another user.

I was wrecking my head on how to properly handle this setup, and I think the most sensible thing to do is to detect files that are NOT owned by current user and make Bvckup ask the user if it should copy the ownership information...

--

On editing filters - http://www.bvckup.com/support/forums/topic_show.pl?pid=265
On backslash - it's not needed

----

@jaden - thanks for the kind words, appreciate them.

"Across remote machines" - do you mean that you would have a UI here, but the actual backup process will run over there? If so, then this is very closely related to running Bvckup as a service. Latter requires splitting Bvckup into the UI process and the engine process (service), and making them talk to each other. Once the split is done (the hardest part), they can technically talk to each other over a TCP connection, so it should be simple to run the UI on a separate box.

However, see the quote in my reply to MerleOne above. There is a principal matter of needlessly complicating the program to make it do something that is useful to a fraction of its user base.

--

With regards to the portable version - yep, as Perry said, it can run without being installed. It can even update itself when run that way. And here is the .zip and here is the thread on the subject :)

----

@Nod5 - I have no idea :) B-vee-kup sounds about right.

----

@Armando - rename/move detection is on my ToDo list, it has been suggested before and I think it's a great feature to have. I will see what I can do... perhaps over the weekend ;)

With regards to further improving speed - there is a number of things that bvckup doesn't currently do, and that I am going to tackle in a bit. Specifically, the bypass of OS cache, using one IO thread per device, etc. That's why it is still in beta.

----

Stephen66515 - Same here, using it mostly for in-LAN chat and RD'ing to my wifey's laptop :) The last thing I wrote for Hamachi was the engine (service) part of the 2.0 client, it was around 2 years ago. The guys who took over are really good, one of the best devs I ever worked with. They took out my easter egg from the About dialog box though... :-|

[KynloStephen66515]

Review is online btw.
-- https://www.donation...ex.php?topic=24273.0 --


Thanks for starting up something great!

[Armando]

@Armando - rename/move detection is on my ToDo list, it has been suggested before and I think it's a great feature to have. I will see what I can do... perhaps over the weekend

With regards to further improving speed - there is a number of things that bvckup doesn't currently do, and that I am going to tackle in a bit. Specifically, the bypass of OS cache, using one IO thread per device, etc. That's why it is still in beta.

-apankrat (October 15, 2010, 08:47 PM)

Nice to hear. Yes, I know it's in beta... But what a beta ! 

I'm still testing it (almost all day today), and even if it's in beta it's probably still faster than most, globally, on most backup jobs.

[sajman99]

Hi, just a couple of questions...

(1) Are Bvckup's cache/database files (ie. in the configuration and logging folder) essential to the operation of the program? AFAICT they must be used for Bvckup to function properly; I can't see how to opt-out of their usage.

(2) Has there been any consideration of an option for direct comparison of files (ie. no reliance on database)? I realize this would likely be a slower method, but I'm just curious.

Regards

[apankrat]

(1) Files in the crc-* folders are the delta copying hashes. If these are deleted, then Bvckup will rebuild them the next time respective file will require copying (and it will copy the file in full). If you don't want these files to be created at all, switch from delta to full copying in backup job's

   Properties => Backup What section => Change Details => Copy What.

The .dat files hold the last seen state of the source directory files - file names, timestamps and sizes. If these are deleted,
Bvckup will copy all files the next time backup is run regardless of the setting in

   Properties => Backup What section => Change Details => What Is Considered Modified

The .ini files are the backup configs. Bvckup does not use any, but one registry entry and keeps all its configuration data in the disk files. So, no, you cannot completely opt-out of using config files, but you can however move them into another directory and tell Bvckup to load them from that directory by using "-c <path_to_config_folder>" command line option.

(2) If you mean bit-by-bit comparison, then - yes, I considered it and just couldn't see it being a popular option to justify the inclusion. Also, considering that delta copying essentially does the same, but at the block level, it seemed a bit redundant.

I am open to be convinced otherwise though

[apankrat]

If anyone's interested, I have reworked the opening post of this thread into a Tell Me More page on the program's website. Had to tone it down a bit though...

[tomos]

I only have one questions: how do you pronounce the name?  "bvackup"? "bee-vee-cup"?

-Nod5 (October 15, 2010, 01:04 PM)

I thought it would simply be pronounced as 'Backup' - that the 'v' is just like an upside-down 'A'