Author Topic: How to deal with syncing multiple versions of a file from multiple sources? (Read 15,098 times)

superboyac · « **on:** July 28, 2011, 04:40 PM »

Problem:
Let's say you have a central server which has your files and documents. This is at home. Now, you also have flash drives and other external storage stuff. While you were away from home, you worked on one of your files, and now you want to update the central server with that file. But you don't want to replace or delete anything because someone else may have modified the central server file also while you were gone. So how can the server always hold the latest file, but also safeguard against deletions or accidental updating.

At first, I would just say to compare timestamps and use the latest one. but there could be the situation where the file in question was modified separately by two different users. So the timestamp wouldn't necessarily really be sufficient. Also, I'm not looking to somehow merge the data from two sources back into one file, that's too complicated for my needs. All I want to do is make sure nothing gets deleted or overwritten.

So I'm trying to figure out the logic of how to have the current file be the latest version, but also not overwrite or delete anything and have the alternate or backup versions stored somewhere else just in case. I'm planning on doing this mainly with a program like Super Flexible File Synchronizer, since it has lots of options for monitoring folders, versioning, and moving things around like this.

I'm curious about your thoughts on this. It's essentially a low-level solution to the multi-user, live collaboration problem. But since there isn't any live collaboration (multiple people editing a file at once), it's more of a synchronizing problem. I'm also trying to avoid services like dropbox and such. These transfers and connections will be direct: either directly connecting to the folders through the network, or sticking in a usb drive.

superboyac · « **Reply #1 on:** July 28, 2011, 05:09 PM »

Here's a flowchart I have so far. Please comment on any holes you notice. One thing I'm debating is whether to have the sync be done live (monitoring folders for changes and immediately syncing), or a once-a-day type of thing.

MilesAhead · « **Reply #2 on:** July 28, 2011, 05:26 PM »

I would search on Version Control System or Document Control System. In software development when you want to edit source code from the depository you have to "check out" the file(s) then check them back into the system when done. The system handles the version control. I haven't used a DCS but I assume it's a similar scheme. The version tree branches when incompatible changes are made. You can always go back and retrieve an old version from the database.

superboyac · « **Reply #3 on:** July 28, 2011, 05:28 PM »

Quote from: MilesAhead on July 28, 2011, 05:26 PM

I would search on Version Control System or Document Control System. In software development when you want to edit source code from the depository you have to "check out" the file(s) then check them back into the system when done. The system handles the version control. I haven't used a DCS but I assume it's a similar scheme. The version tree branches when incompatible changes are made. You can always go back and retrieve an old version from the database.

Thanks man, I knew there must be a common term for this. now I can search and read about it.

superboyac · « **Reply #4 on:** July 28, 2011, 05:30 PM »

Before I get too far into it, can someone recommend a good software for document control system? No cloud services, please.

superboyac · « **Reply #5 on:** July 28, 2011, 05:31 PM »

Seems like SVN is the most popular choice here...

superboyac · « **Reply #6 on:** July 28, 2011, 05:57 PM »

Bazaar is looking appealing to me. It seems to have a nice GUI. All the others are very command line-ish, which I hate.

kyrathaba · « **Reply #7 on:** July 29, 2011, 12:15 PM »

Haven't checked out Bazaar. May be great, dunno. I have worked with SVN using TortoiseSVN (FOSS). Even I was able to get up to speed on it in a few minutes' time, so you should find it a quick-study, should you choose to go that route.

Stoic Joker · « **Reply #8 on:** July 29, 2011, 09:00 PM »

Once you start getting too many versions of a fine it's almost worse than losing said file. As it becomes an ordeal to figure out which is the correct latest file - With the changes you're looking for.

I'd prefer to keep it simple and leverage the previous version feature that is already built into Windows.

superboyac · « **Reply #9 on:** July 29, 2011, 09:17 PM »

Quote from: Stoic Joker on July 29, 2011, 09:00 PM

Once you start getting too many versions of a fine it's almost worse than losing said file. As it becomes an ordeal to figure out which is the correct latest file - With the changes you're looking for.

I'd prefer to keep it simple and leverage the previous version feature that is already built into Windows.

Well, i don't plan on doing this with all files. There are certain files that I will be regularly modifying both at home, at work, and on the run (think a notetaking application that captures clips of stuff you come across). So I need a way to handle that with the minimal amount of manual tuning. Let's say I added some notes to a database on my USB drive, then I go home and capture a couple of notes on the desktop...then I remember I had a couple of new ones on the USB...how do I combine everything?

Shades · « **Reply #10 on:** July 29, 2011, 10:01 PM »

Git (open source, portable DVCS system) is supposed to be the champion of merging changes. Say your original set of files is your home server > you 'check-out' to your USB stick > make changes on the USB files > merge these changes back when at home. Changes that are made at home to these files in the mean time will show up as well when you take a look at the changed file(s) on your USB stick and home server with SmartGit (free for personal use, portable).

That makes it easier to see/decide which changes you want to keep.

Armando has a very informative DVCS thread here on DC, including links to all the software and manuals you will ever need to become very proficient with tools like Git/DVCS. In the same thread alternatives are reviewed as well.

superboyac · « **Reply #11 on:** July 30, 2011, 05:12 AM »

Sweet! Thanks, Shades. I'll have to check out Armando's thread, he's always extremely helpful with this sort of thing.

superboyac · « **Reply #12 on:** July 30, 2011, 05:14 AM »

Armando's thread:
http://www.donationcoder.com/forum/index.php?topic=25765.0

Ath · « **Reply #13 on:** July 30, 2011, 08:32 AM »

A real pain will be to merge data from binary files like Office documents and databases. All the usual (D)VCS systems are based on merging text-like documents, such as source files of a programming language, not binary data. Some of them have extensions/plug-ins to read/compare/merge Word or pdf documents, and that's often the best you can get. For comparing/merging databases there's a whole different SW industry, (I've not investigated 'cause I don't need nor want to go there).

I/we usually do the merges we need to do with an IDE like Eclipse and/or Visual Studio and their available plug-ins (for SVN, that's our central VCS of choice), and so far there have not really been merging issues, not even with international development teams working on the same repository.

superboyac · « **Reply #14 on:** August 01, 2011, 02:06 PM »

Quote from: Ath on July 30, 2011, 08:32 AM

A real pain will be to merge data from binary files like Office documents and databases. All the usual (D)VCS systems are based on merging text-like documents, such as source files of a programming language, not binary data. Some of them have extensions/plug-ins to read/compare/merge Word or pdf documents, and that's often the best you can get. For comparing/merging databases there's a whole different SW industry, (I've not investigated 'cause I don't need nor want to go there).

I/we usually do the merges we need to do with an IDE like Eclipse and/or Visual Studio and their available plug-ins (for SVN, that's our central VCS of choice), and so far there have not really been merging issues, not even with international development teams working on the same repository.

Yes, very true. This is something I'm struggling with. Here's my problem: i don't really care about merging and subversion control all that much. It's not that I need everything to merge properly. I just want to figure out a way to manage this workflow where I do some stuff at home, some stuff at work, some stuff on a usb stick. I think I can get away with my initial idea which didn't have any kind of subversion control, just some file synchronization tweaks. If I bring my usb to the homebase, the synchronizer will check to see which file is newer, and put that in the working directory. Any older files won't be replaced or deleted, just moved to another location (like a subfolder). That's all i want. I don't think I need anything more than that.

f0dder · « **Reply #15 on:** August 01, 2011, 02:55 PM »

The short and implementable-in-real-life answer: cover your eyes, and run away screaming, arms flailing.

f0dder · « **Reply #16 on:** August 01, 2011, 03:11 PM »

The longer reply...

Quote from: superboyac on August 01, 2011, 02:06 PM

i don't really care about merging and subversion control all that much. It's not that I need everything to merge properly. I just want to figure out a way to manage this workflow where I do some stuff at home, some stuff at work, some stuff on a usb stick.

And what do you intend to do when (not if!) things get out of sync? This is a pretty darn bad workflow unless you want to end up in a royal mess... I'm fully aware that having strict procedures for how/when you edit files is also somewhat of a pain, but believe me - it's far less than the "Oh fsck, which is which?" mess you can end up in otherwise.

Also, timestamps are a notoriously bad way to handle things. Say you do some important edits to "SuperStory.doc" on your workstation, then have to toddle off for a weekend trip. There, you do some other minor work on the same base SuperStory.doc on your laptop (which doesn't have the changes made on your workstation). Your laptop has the most recent timestamp, but your older-timestamped workstation copy has the most important edits.

What you need to realize is that there isn't an end-all-be-all system that can handle what you want across all possible file types. It doesn't exist, not even if you're going to throw a gazillion dollars at enterprisey systems. What you'll find will either be too generic or too specialized.

(D)VCS platforms tend to handle binary data poorly - no go for .doc files or images. It's a (very) viable solution if your documents are in sane textual formats (latex, docbook, svg, html, ...) - but if you mainly deal with binary files and can't switch to something better, it's not really a solution either.

Don't even think about using subversion - it's OK when you need to deal with legacy systems, but it has numerous issues and should be avoided, considering there's much better modern solutions. Bazaar, Mercurial, Git, et cetera.

Lashiec · « **Reply #17 on:** August 02, 2011, 10:21 PM »

Quote from: f0dder on August 01, 2011, 02:55 PM

The short and implementable-in-real-life answer: cover your eyes, and run away screaming, arms flailing.

The shorter answer: use Google Docs or SpiderOak

Stoic Joker · « **Reply #18 on:** August 03, 2011, 11:55 AM »

Quote from: f0dder on August 01, 2011, 02:55 PM

The short and implementable-in-real-life answer: cover your eyes, and run away screaming, arms flailing.

$:-\$ ...How do you flail and cover your eyes at the same time?

Are you Zaphod Bebelbrox?

f0dder · « **Reply #19 on:** August 04, 2011, 07:03 PM »

Quote from: Stoic Joker on August 03, 2011, 11:55 AM

Quote from: f0dder on August 01, 2011, 02:55 PM
The short and implementable-in-real-life answer: cover your eyes, and run away screaming, arms flailing.
$:-\$ ...How do you flail and cover your eyes at the same time?

Cloth, or perhaps tar - there's lots of possibilities ;P

Quote from: Stoic Joker on August 03, 2011, 11:55 AM

Are you Zaphod Bebelbrox?

Humm, didn't he have multiple heads rather than arms?

Stoic Joker · « **Reply #20 on:** August 04, 2011, 08:14 PM »

Quote from: f0dder on August 04, 2011, 07:03 PM

Quote from: Stoic Joker on August 03, 2011, 11:55 AM
Are you Zaphod Bebelbrox?
Humm, didn't he have multiple heads rather than arms?

Yes to both, he had 2 heads, and 3 arms. The third arm was added for the sole purpose of cheating at a game whose name I forget.