topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Thursday March 28, 2024, 3:38 pm
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Superboyac's backup strategy revisited (revised for 2011)  (Read 9761 times)

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
I have discussed backup strategies here a lot, and here I go again.  Just to remind everyone, mouser wrote a detailed backup guide several years ago:
https://www.donation...ckUpGuide/index.html

and I wrote my own limited guide a little after that:
https://www.donation...dex.php?topic=7940.0

Now, I am going to write some more.  I'll eventually publish a cleaned up version of everything on my website once I figure everything out.  The stuff below are my ramblings based on several discussions I've had with individuals and even people here at DC.  I still have some questions and issues to sort out, but I think I have an overall handle on things now.

I want to drastically improve my current backup solution.  I have a double redundancy backup going on right now.  Meaning, I have a hard drive where I keep my personal files (no OS or installed programs, that's on another drive).  I mention that because all the files/folders are standalone, and can be moved anywhere without a problem (no drivers or OS required, they're just files).  Anyway, when I built this backup system, I used 1 TB drives.  So I have two backup 1TB drives backing up that data drives.  3 drives total, which equals double-redundancy.

Now the problem is that I'm running out of space.  When that happens, I clean out the less important stuff by burning it onto a DVD.  But this is a klunky way of doing things.  I would rather have everything on hard drives and backed up that way.  That means that I need to purchase additional hard drives.  However, I already have 5 drives on my desktop (internal+external).  Adding more at this point would be a little bit much.

That's why I'm building a server.  This issue along with my desire to put ALL my data on hard drives means I've stepped across the simple desktop boundary and into server territory.  The knee-jerk reaction at this point is to buy a NAS thing and be done with it.  But I like to do things my way, and I like to do things a little on the extreme side.  I realize that there are more simple and affordable solutions to this that are perfectly adequate.  But I'm going to do it the hard way and get a system that I like better.

First question that comes to mind: how many hard drives do I need?  Well, how much data do I have currently?  Right now, I have about 1 TB of data.  But I also have hundreds of burned DVD/CD that I eventually want to stick back onto hard drives.  Plus, I want to take my entire movie collection (home movies, DVD's, etc.) and put in on the hard drives.  The movies are huge (when uncompressed using makeMKV) so that will add significantly to my size requirement.  Altogether, taking future growth into account, I'm going to plan for backing up up to 4 TB of data.  Yes, I know it's a lot, but it makes sense, especially with all the movies.

The next question is the method of file backup.  The first "track" backup (mouser's term!) will be file synchronization.  I really prefer file syncing because it means I can use the files very easily, and I can just grab a hard drive and plug it into another computer and start using it without any extra steps.  Image backups, on the other hand, are more difficult to use because you have to extract the files, you need additional software, it's not easy to use with other computers.  That's why I love file syncing.

I have a double-redundancy philosophy with file syncing.  It would take a pretty rare circumstance to simultaneously wipe out data from three different hard drives.  Also, I very narrowly avoided losing my original data AND the backup data a few years ago when I was doing single-redundancy (long story).

At this point, the most pressing issue is that backing up 4TB of data with double-redundancy is a LOT.  Assuming I use 2TB drives for each set, I would need 2x3=6 hard drives (2TB each) to accomplish this.

The next track for a backup strategy is doing image backups.  While file syncing is great for portability and convenience as far as accessing individual files/folders, it is not that suitable for versioning and OS/programs backup.  I'm not that concerned about backing up the operating system or installed programs because you can always reinstall that stuff.  I'm way more concerned about my personal data.  What I desire from images that I don't get from file syncing is versioning.  Meaning, let's say I deleted or modified a file from last month, and only now am I realizing that I wish I had the original file back.  Versioning keeps track of all these changes and you can recover them from images.

Versioning, from what I've tried, can be done in two ways: using images, or using archived file sets (rar,zip).  I used to do it the archived file way.  I've tried the versioning support in SFFS which does versioning by appending dated suffixes to files.  It's not an elegant way of doing it.  I've also tried programs like Genie.  Programs like Genie and Backup4All can use archive formats to manage their versions.  If you need to recover an old file, it extracts it from a zip/rar file (unless you store them uncompressed).  But doing it this way is more like a hybrid between file syncing and images.  I didn't like it very much, and that's why I'm going back to images.

You might be asking why I'm so hesitant to use images.  As mentioned above, one of the reasons is that I don't really care about backing up my OS or programs, so I don't need something like images that retains all the interrelated files/drivers/OS system files.  If you haven't noticed, I REALLY like having portability with my files.  If it were up to me, everything would be portable: the OS, programs, everything.  So that's why I struggle with this part.

Anyway, so I've decided to use images for versioning because it's the best way right now.  Since versioning adds the variable of time to the pot, I have to be more thoughtful about the backup schedule.  With file-syncing, since I'm mirroring the files/folders, time is not an issue.  They are just copying files, and it doesn't really matter when it happens.  With versioning, I want the images to give me the ability to go back in time.

I've really struggled with the best way to set this schedule up the past year.  Then I saw Apple's Time Machine program.  I really liked it, so I'm going to model my imaging setup to mimic that.  So what does that mean?  Here's the explanation (from Wikipedia):
"Time Machine saves the hourly backups for the past 24 hours, daily backups for the past month, and weekly backups for everything older than a month"
That's what I want to do.  I don't really know how to set up image programs to do this right now, but I'm pretty sure it can be done.  What I need to do is set up a 3-track imaging approach: one will do the hourly backups, the second will do daily backups, and the third will do the monthly backups.  My question is, do all these tracks need to be tied together somehow, or are they independent of each other?  I don't know right now.  Obviously, Apple's program is seamless.  I'd like to be able to do the same, but whether or not that's possible remains to be seen.

Once again, how many additional hard drives will I need for this imaging stuff?  Well, I'll be backing up the same 4TB of personal data, but I will also throw in the OS/programs.  I'm still going to consider it as 4TB total.  Once again, I'm going to do this with double-redundancy.  This means I will need an additional 4 drives (2TB each).

If you're keeping count, I now have 10 drives total.

Back to the server.  I need some kind of server rack to hold all these drives.  I'm not familiar with all the lingo yet (rack, enclosures, 2U, 4U, etc.) nor do I care.  I just need a box that houses the drives, the computer, etc.  I'll probably get some kind of mid-height (no more than 4') rack tower for all of this.  I need something like a Dell Poweredge unit to hold the drives.  Allowing for some future growth, I'd like it to hold 15-20 drives.  Then I need some kind of unit for the motherboard and all that.  Then some kind of rack monitor and keyboard, unless I choose to access it remotely.

This is when people normally start talking about RAID setups, and I just don't get it.  I don't think I need RAID.  There are many things I don't like about RAID.  Firstly, I don't need the speed.  Secondly, It would at least double the amount of drives I would need.  All those drives, if RAIDed, would need identical duplicates to do all that mirroring and building and such.  It just sounds like a big headache to me.  It's something I'm unfamiliar with, and uncomfortable with.  I've had issues in the past trying to deal with RAID related issues with motherboards, bios, hard drive configuration.  It's just a headache.  And I don't think it really offers much to my backup strategy.  If it were simple and affordable, I'd consider it.  But it's just a headache in every way.  I hate listening to people talk about all the different RAID flavors, it annoys the shit out of me.  I don't think most of them know what they are talking about.  I'm convinced of one thing: it's not a true backup strategy.  It's only a "kind of, sort of" backup strategy.  So unless someone can convince me otherwise, I'm just going to do this using independent hard drives and software.  No one has ever convinced me yet, and I've asked a lot of people.  Like I said, I think there are a lot of people out there who think they are RAID experts, but when I talk to them, it's clear that they don't know enough to answer my questions.

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #1 on: May 19, 2011, 03:05 PM »
RAID isn't a backup strategy per se. It's a strategy for data redundancy and recovery in the event of hardware failure. RAID can figure into a backup strategy - but it's not a substitute for backing up.

RAID is also oversold. IMO it's only really useful for use on a server when mirroring (RAID1) the drive holding the operating system. That provides fail-over in the event one drive unit dies.

The only problem with this strategy on a home server is the quality of the RAID controller card or subsystem on the mobo. Quality and reliability of these "consumer" or "semi-pro" controllers varies a great deal from the more expensive and better designed "enterprise" controllers.

If your RAID controller doesn't have it's own CPU and RAM, it's "consumer grade" - and you have been warned. I've seen these types of controllers malfunction and corrupt both drives in a mirror far more often than I've seen the drives themselves fail. If you do use inexpensive RAID controllers to mirror your boot drive, do yourself a favor and take periodic recovery snapshot images. Because sooner or later - you're gonna need them.

I'm seriously thinking the next home server I build is going to be configured to use laptop-type 2.5" form factor drives. They're smaller, more energy efficient, and shock resistant than regular SATA drives. And there are some new home servers already set up to use them which look quite promising. I'll likely keep minimal amounts of user data (MyDocuments, etc.) on it, and use an external array for most of my heavy (i.e. libraries and archive) storage requirements to avoid heat issues in the server case. Something from Synology, Sonnet Technologies or Granite Digital will probably be what I use for that part.

Check out the following sites for reviews and product info:

ServeTheHome  :-*
SmallNetBuilder  :-*
MyHomeServer
WeGotServed

 :Thmbsup:
« Last Edit: May 19, 2011, 03:07 PM by 40hz »

JavaJones

  • Review 2.0 Designer
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 2,739
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #2 on: May 19, 2011, 04:27 PM »
Agree with 40hz, RAID isn't backup, and I wouldn't recommend it for the home user (even a power user). You don't need to worry about it.

Now, to address your specific remaining issues/questions:

First, regarding your desire to do versioning with "imaging", this is certainly possible, however if the only reason you want to do imaging is for versioning, then it's probably overkill and not the best approach just to get versioning. If you want good versioning support with backup, try CrashPlan. The software is free if you don't use the Pro or online features, but it's also worth the price if you do need e.g. backup sets (a Pro feature). There are other backup programs that also do versioning, I just find CrashPlan to be a nice, comprehensive option with good versioning support. It works similar to TimeMachine in that it saves more frequent versions for more recent times. The version saving is customizable as well. If you do decide to go with imaging, make sure you select an app that does on-the-fly/"live" imaging (so you don't have to shut the machine down) - most do these days - and more importantly that it does in fact support versioning. Not all imaging apps do versioning, it's a more common feature for non-image based backup tools.

Note also that if you're doing versioning, your basic space calculations may be way off, depending on how many large files are changing.

The next big question is rack mount. If you have space for one and a desire to add more equipment over time, then it *may* make sense for you. Just keep in mind that the rack itself will cost, and most equipment designed for racks is interprise-grade, generally more expensive and higher-spec'ed than you probably need (e.g. redundant power supplies). I personally wouldn't bother with a rack. That being said if you really do need the number of drives you have spec'd, you may need to.

If you do go rack mount, you'll be looking at something like this, some more bare options or these guys. You can see how expensive it can get quite quickly.

Also, if you don't already have one, be sure to get a good UPS and configure it properly on the machine so that it will gracefully auto shut down before the backup power runs out. This will help minimize the chances of any failure due to power outages, surges, etc.

Finally, if you're going to all this length and expense to get a backup system with versioning, you should seriously consider the issue that all your data is still in one physical space. If there is a fire or other natural disaster, or even if some careless person spills water or something on the system, it could at the least mean needing to pursue data recovery, and at worst could mean your careful (and expensive) backup strategy is for naught. Consider options for off-site backup. It may seem prohibitive with that much data, but I have a similar amount and am about to start using CrashPlan's online service. They offer a backup seeding service for a reasonable price to get you started with the majority of your initial data, after that it's just maintenance so as long as you have a reasonable Internet connection (reasonable outgoing bandwidth that is), it should be fine.

- Oshyan

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,646
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #3 on: May 19, 2011, 06:33 PM »
Then I saw Apple's Time Machine program.  I really liked it, so I'm going to model my imaging setup to mimic that.  So what does that mean?  Here's the explanation (from Wikipedia):
"Time Machine saves the hourly backups for the past 24 hours, daily backups for the past month, and weekly backups for everything older than a month"
That's what I want to do.

Sounds alot like the Previous Versions/Shadow Copy Service that's built into Windows. Can be configured to do snapshots daily, weekly monthly, or every 5 minutes (not recommended), as you like. Default is twice a day.

tomos

  • Charter Member
  • Joined in 2006
  • ***
  • Posts: 11,959
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #4 on: May 19, 2011, 06:45 PM »
Then I saw Apple's Time Machine program.  I really liked it, so I'm going to model my imaging setup to mimic that.  So what does that mean?  Here's the explanation (from Wikipedia):
"Time Machine saves the hourly backups for the past 24 hours, daily backups for the past month, and weekly backups for everything older than a month"
That's what I want to do.

there's a few windows backup apps that do that - just dont ask me names :-[


(have backup4all introduced something along those lines? I've lost track of it lately)
Tom

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #5 on: May 19, 2011, 07:28 PM »
Thanks Java!  That's a lot of food for thought.  Also, 40..thanks, as always!  (One day I'm going to steal your knowledge database wherever you keep all your links and notes and stuff.  I know you have something somewhere...)

Looks like you guys have given me some things to investigate.  I will do it gladly.  I'm not 100% sold on the imaging thing, so if there are alternatives to achieving the Time-Machine like versioning, I'm all for it.


JavaJones

  • Review 2.0 Designer
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 2,739
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #6 on: May 19, 2011, 07:33 PM »
I would venture to say that full system imaging is in fact more the "alternative to achieving" and *other* methods are actually more common for achieving versioning. If full system images aren't important for immediate restore to the current OS state, then they may be more hassle than they're worth.

- Oshyan

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #7 on: May 19, 2011, 08:22 PM »
I would venture to say that full system imaging is in fact more the "alternative to achieving" and *other* methods are actually more common for achieving versioning. If full system images aren't important for immediate restore to the current OS state, then they may be more hassle than they're worth.

- Oshyan
That's my belief also.  I'm checking out your Crashplan recommendation.  It's pretty good!  My initial impression is that it seems a little TOO simple for me, but that's just an assumption.  If it can do everything I want, that's all I need.  I like the window where you can choose which file to restore (from the history of that file).  Pretty good stuff.  I'd like to explore some more alternatives and think about it, but it seems to be a solid choice.

JavaJones

  • Review 2.0 Designer
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 2,739
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #8 on: May 19, 2011, 09:15 PM »
Unfortunately CrashPlan does not offer a direct sync option. If it did it would be all the backup software I need. So my own backup strategy involves a separate sync app and then CrashPlan for local backup, versioning, and cloud backup.

- Oshyan

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #9 on: May 19, 2011, 10:43 PM »
Hey!  Oops!Backup seems to be excellent:
http://www.altaro.com/

Maybe that's the one!

JavaJones

  • Review 2.0 Designer
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 2,739
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #10 on: May 19, 2011, 10:46 PM »
I've heard good things about Oops!Backup but haven't tried it myself.

- Oshyan

ilovekonoka

  • Supporting Member
  • Joined in 2009
  • **
  • Posts: 11
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #11 on: May 20, 2011, 02:27 AM »
I think Oops Backup is very good and it seems quite fit for superboyac's plan. I've been using it for quite a while and it has saved me a few times. I set it to backup every 1 hour if there are any changes, and keep older versions for up to 3 months. My whole data is about 50GB and the total backup size is now about 150GB.

I'm showing some usage statistics below if that can help

[attachthumb=#1][/attachthumb]
[attachthumb=#2][/attachthumb]
[attachthumb=#3][/attachthumb]

phitsc

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 1,198
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #12 on: May 20, 2011, 02:58 AM »
But how can you ever name a professional product Oops!Backup :o

Armando

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 2,727
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #13 on: June 21, 2011, 01:45 PM »
Oops Backup has some issues (mostly space and RAM issues). I don't know if they've been ironed out, but you can do a search for related threads.

I agree with others about imaging. Good for OS backup, with versions, etc. but not so great for everything else.

If you remember, I posted about my own strategy a while ago (https://www.donation...99.msg58886#msg58886). It hasn't changed much as my requirements are still the same.

1 a- Sync my ALL my files (not the OS) frequently (everyday) to 2 different 1TB hard drives.
   b- Versioning of everything**
(I use Syncback for that. Was using bvckup, but still waiting for some issued to be ironed out to go back to it)

2 a- Backup off-site : means that I "send" away one of my hard drive elsewhere once a week (I just swap harddrives -- parents keep it in their house.),
   b- I also use Spideroak as an extra cushion monitoring work stuff I absolutely can't loose (where even an hour would be super annoying). I don't need to have absolutely everything synced using some "online plan" -- just a few folders.

3- Image my OS once a month (and keep versions)... or as needed.

** I Archive file versions on older hard drives (those who got too small) or DVDs.

All this takes some thinking, but not that much.

It's extremely easy to maintain and cheap. The only thing I check once in a while is if Syncback is doing its job properly. And it always did/does...  :)

The only part that is a bit trickier is the off-site hard drive swapping -- I sometimes forget. But the worst that could happen if my house burnt is that I'd loose a couple weeks of probably unimportant stuff since the most important things are synced through Spideroak anyways...
« Last Edit: June 21, 2011, 11:01 PM by Armando, Reason: Changed \"system\" for OS + a few minor word changes. »

Armando

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 2,727
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #14 on: June 21, 2011, 03:53 PM »
Here's the Oops!Backup  link :

I re-read parts of the thread again and it certainly didn't convince me to try oops again. They'd really need to work on some of the mentioned problems... For smallerish files, it should be okay though.

===

It's interesting for me to see how I tried so many options during the last few years and just went back to my original "no fuss" solution -- full proof, simple, efficient. The only things I added to the mix are online syncing ingredients like SpiderOak (not always completely reliable unfortunately...).

I don't think I'll come back to that "backup struggle" until next summer, when I'll need extra storage... Way too time consuming... Need to get some stuff done!!!!  :'(
« Last Edit: June 21, 2011, 10:56 PM by Armando, Reason: (missing a word) »

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #15 on: June 21, 2011, 09:50 PM »
Here's the Oops!Backup  link :
I re-read parts of the thread again and it certainly didn't convince me to try oops again. They'd really need to work on some of the mentioned problems... For smallerish files, it should be okay though.

===

It's interesting for me to see how I tried so many options during the last few years and just went back to my original "no fuss" solution -- full proof, simple, efficient. The only I added to the mix are online syncing ingredients like SpiderOak (not always completely reliable unfortunately...).

I don't think I'll come back to that "backup struggle" until next summer, when I'll need extra storage... Way too time consuming... Need to get some stuff done!!!!  :'(
Thanks Armando, I'll have to read that.  Yeah, I'm struggling with the same issues.  I'm surprised about Oops because it seems like they have good customer support and are diligent with addressing issues.  But I trust you more than the other stuff I read!

Armando

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 2,727
    • View Profile
    • Donate to Member
Re: Superboyac's backup strategy revisited (revised for 2011)
« Reply #16 on: June 21, 2011, 10:54 PM »
You could still test it though. But test it in a realistic setup, with databases and other bigger files. This is where the problems started -- versions pile up and take a huge amount of hard drive space. And then it's not that intuitive to get rid of versions etc. Well, it wasn't... Don't know if it is now.

I'll just quote J-MAC from the other thread (I hope he doesn't mind...) :

Also it has difficulties - big difficulties - deleting old backups. My backup storage grew a little too big too fast for my liking so I wanted to reduce its size. If you delete the oldest backups from within Oops it slows to a crawl. E.g., deleting less than a week's worth of backups Oops took over 20 minutes just to select the old backups! Then a couple hours - running at 75 to 85% CPU trying to delete them. It was unresponsive for so long I finally gave up and killed the process - and the service, too - and then just went into DOpus and manually deleted the files and folders. Manually. Took all of < 1 minute.

Then last night I noticed that the Oops storage area had increased by an additional 145 GB! I found a folder named "Recovered Files" in the Oops Profile folder. Why? I don’t know yet. I deleted that too.

It just doesn’t act like most other backup programs I have used, and it doesn’t allow you to freely change, delete, pause, etc.; all features that Altaro really needs to implement in order to make it a usable product IMO. I'm awaiting the next reply from support to see what I do from here.

Personally, I like having as much control as I can... At least in that area of my life.  :)