What is your preferred server OS for home use? And Why?

[JavaJones]

Just to clarify, I wasn't suggesting SB get a bunch of NAS units. The whole conversation has centered around a single "server" machine, I was just assuming more drives in the machine to handle redundancy, as SB himself had suggested. I think when I make my 2nd blog post about my backup strategy some of my thinking on all this will become more clear. My opinion has changed a bit since my big drive failure.

Unfortunately there's no easy way around how to *fully* back up 10+TB of data, I agree. Fortunately I don't have that much data I really need to back up. It sounds like SB maybe does, so that's a conundrum. I could easily suggest a pretty reasonable backup strategy that would be "secure" and "redundant". It would just take a lot of hardware, heh. I wouldn't normally see RAID as necessary, but it might play a part solely for the simplicity it brings in addressing large contiguous disk space. Otherwise it just adds cost without really adding a worthwhile level of redundancy (IMHO, I'll just leave it at that).

As far as the idea of having to determine what's important and what's not, I think this kind of has to be a part of any backup strategy. For example some people (like myself) don't really care about backing up the whole OS, but want all the data, and that in itself is a decision about data importance that's made. Going the next step and realizing that you don't necessarily *need* triple redundancy on your archive of ripped DVD movies since you have the originals is a reasonable and important thing to do. It saves you having to back up lots of large data. I've made this call about my media archive, because I can always re-rip or re-download something. I can never re-take a photo and re-writing a document will never be the same, so that stuff is worth triple redundancy if possible.

If you really don't want to think about it and have the money and hardware space, just get 3x the number of drives you need, use basic sync/mirror for one data set, and a versioning backup program (like CrashPlan) for the other. That way you have a normal copy of the data that doesn't depend on a backup app functioning correctly (the sync/mirror) and a versioned backup that should help in the case of e.g. viruses, etc. Of course this isn't a truly bulletproof solution, first because there's no off-site backup, and second because there's only one versioned setup and if that fails and you need versioning, you may be hosed. But I consider that a pretty outside case.

Off-site backup of 10+TB of data is sadly not terribly practical. But again there are solutions to it if it's truly important. That's a situation where I really do think an external NAS-type box (but actually a locally connected one, e.g. eSATA, not network-attached) would make sense. You can get external RAID boxes that support 5 or more drives, drop 2TB units in there, and just RAID0 it so you have a big contiguous drive to back up to. RAID0 is inherently more dangerous of course, but since this is just 1 of 2 or 3 backup locations, it's ok. It's really just there for catastrophic events (e.g. your house burns down, taking your other 2 backups with it). If you have the additional drive space to do so, might as well RAID5 of course.

Note I am only suggesting RAID here because it makes it easy to address it as one big unit, not because of redundancy. I'm not sure how difficult it really is to externally address *multiple* SATA drives through eSATA, though I know it's possible (and there have been some discussions here about it). If it's possible with some of the existing external SATA drive enclosures, then it might be preferable just to have them as an additional sync target, not in a RAID but as individual drives.

Anyway, if we assume 10TB and that securely and redundantly backing up *all* of it is desired, I'd therefore suggest the following, summarized from above:

• 1 system with 10 2TB SATA drives
• An automated sync or mirroring between 1 set of 5 internal 2TB drives to the other set
• 2 external drive enclosures, one with 5x2TB drives the other with more if possible (this would be the incremental backup drive)
• Use one external enclosure in simple disk mode if possible, doing an additional sync of all drives to the external unit once a week, then take it off-site (e.g. to a storage unit, safe deposit box, etc.)
• Use the other external enclosure in RAID (RAID5 if possible) and use an incremental backup process on the internal drives targeting the external unit

This setup is pretty overkill and expensive, but will give you triple data redundancy *with* off-site backup *and* versioning. If you don't care about the off-site backup, then just remove that part from the equation and simplify.

- Oshyan

[Stoic Joker]

It's a question of static vs. dynamic data. Backup frequency is dependant on how often critical changes are made.

Example:
I have a client with 300GB of important business data, and a 36GB tape backup drive. But, 90% of the data is historical (e.g. never changes). So... The have 2 servers, one new, and the old one it replaced.

Server 1 (the new(er) one), runs SBS2k3, and stores all their currently active stuff. It also has the tape drive in it. Backup of their active data, System State, AD, etc. require all of one tape, and sometimes part of a second. Exchange (also critical to backup) just flat won't fit...without going to a 3 tape a day rotation - Which is insane. So...

Server 1 is RAID5, That's the primary line of defense for the OS in the event of a disk failure. secondary line (if 2 disk failure, etc.), is the daily backup of AD and System State data. The complete configuration is also heavily documented so a brick level rebuild can be done quickly if necessary. The other business data is backed up fully Monday through Thursday when a 2nd tape can be added in the morning if necessary (this is the data that changes most frequently).

Exchange is backed up on Fridays only by itself, because it handily fits on one tape, and nobody is there on Saturdays to add a second tape. otherwise a daily business data backup would always fail (timeout) on Saturday morning if it was run. So there I've staged when what was backed based on a heriarchy of what was most likely to change to maximize the highest number of changes that could be saved.

Now the bulk of the static historical data on server 2 (also RAID5) is backed up once a month, to one of a pair of 1.5TB USB drives. This is because the data on it only changes when the file handling policy declares that something is "old enough" and is manually pushed to the server. This is done in the last few days of the month, because the USB drive backup is scheduled on the 1st of every month.

That way if - by some odd strike of bad luck - something went wrong big time ... the only unbackedup data that could be lost, would be the most current, and therefore easiest data to recreate.

All of the backup operations are logged, and a full report of the results are Emailed to both myself and the business' owner every morning. That way if something does go wrong with the backups... We Know!

[Stoic Joker]

Damn JJ we're turning into a couple of long winded bastards here ain't we? I'm guessing you type alot faster than I do 'cause ya beat me ... So I'll have to read yours in a minute.

 

[JavaJones]

Hehe, I have a pretty decent typing speed. I'm definitely long-winded, which is probably part of why I've become a fast typer.

Personally, though I used to use tape backup more than a decade ago, I just don't see myself ever going back there given the low cost of drive space and speed advantages (both for backup and restore). I know it's still widely used by enterprise though.

- Oshyan

[Stoic Joker]

Personally, though I used to use tape backup more than a decade ago, I just don't see myself ever going back there given the low cost of drive space and speed advantages (both for backup and restore). I know it's still widely used by enterprise though.

-JavaJones (January 06, 2011, 10:55 PM)

I'm with Ya, I'm beginning to transition to USB drive backups myself now that the small easy to carry 2.5" drives are over 500GB and under $100. However the setup I described above was deployed 5 years ago when that wasn't the case. So tape made sense back then.

My point for SB was how the backups can be staggard to maximize coverage and minimize futzing arround at the same time. By organizing the data based on (static vs. dynamic) change frequency. And I agree that not everything really needs to be backed up if it's already stored as a hard copy, or just really not that critical (we all have a few files like that...).

[f0dder]

--The one thing I am worried about and will probably spend the most time thinking about: how to prevent BAD backups and only have good backups.  Let me explain.  Let's say I have a hard drive, and I back it up with double-redundancy using two more hard drives and syncing (SFFS).  Now, let's say the original drive got infected with a virus that got into a bunch of my files.  Soon after, that virus will be backed up to three places.  And now I don't have an original GOOD version anywhere!  How do you prevent that?  It doesn't help if you back up with multiple redundancy if they all have the same BAD files.

-superboyac (January 06, 2011, 09:14 PM)

If you become infected, you should expect all files on all disks on your system to be infected... but (silent) file corruption not caused by virii is another situation.

You could do a staggered setup, but that's going to require a fair amount of disk space, and will probably be a maintenance nightmare.

I don't know a Windows equivalent, but rsnapshot is very nifty. It does incremental backups, but not in the traditional sense... take a look

[Stoic Joker]

--The one thing I am worried about and will probably spend the most time thinking about: how to prevent BAD backups and only have good backups.  Let me explain.  Let's say I have a hard drive, and I back it up with double-redundancy using two more hard drives and syncing (SFFS).  Now, let's say the original drive got infected with a virus that got into a bunch of my files.  Soon after, that virus will be backed up to three places.  And now I don't have an original GOOD version anywhere!  How do you prevent that?  It doesn't help if you back up with multiple redundancy if they all have the same BAD files.

-superboyac (January 06, 2011, 09:14 PM)

If you become infected, you should expect all files on all disks on your system to be infected... but (silent) file corruption not caused by virii is another situation.

-f0dder (January 07, 2011, 06:00 AM)

Realistically how likely of a threat is that (I've been wondering about this for a while)? It seems like the bulk of the EvilDoerWare these days only goes after the shell and your wallet/identity (not necessarily in that order). There really hasn't been a good file eating virus (that I recall) in the wild since Snow White almost a decade ago. The SBD file corruption seem a more (real?) likely threat these days to me (But I'm on the fence).

Mind you I'm not arguing the point, I really am just curious about your thoughts on this matter.

[4wd]

I don't know a Windows equivalent, but rsnapshot is very nifty. It does incremental backups, but not in the traditional sense... take a look

-f0dder (January 07, 2011, 06:00 AM)

Dammit!  I'm positive I found one on the net recently that was based upon this blog article: An intelligent backup system for Windows (4 part article)

Do you think I can find it again....nope!

Found it: DataSafe Backup


Lucky I never throw away anything I download

[f0dder]

If you become infected, you should expect all files on all disks on your system to be infected... but (silent) file corruption not caused by virii is another situation.

-f0dder (January 07, 2011, 06:00 AM)

Realistically how likely of a threat is that (I've been wondering about this for a while)? It seems like the bulk of the EvilDoerWare these days only goes after the shell and your wallet/identity (not necessarily in that order). There really hasn't been a good file eating virus (that I recall) in the wild since Snow White almost a decade ago. The SBD file corruption seem a more (real?) likely threat these days to me (But I'm on the fence).

-Stoic Joker (January 07, 2011, 06:58 AM)

Dunno if you're likely to run into a .exe infector these days, like you I've got the feeling (though not scientifically backed!) that they're a thing of the past.

There's been several examples of ransomware the last few years, though, and those go after all your files, not just executables...

[40hz]

I don't know a Windows equivalent, but rsnapshot is very nifty. It does incremental backups, but not in the traditional sense... take a look

Had to dig into our KB for this one: There is something called cwRsync that will allow you to easily integrate the rsync daemon with various flavors of Windows. Catch is the Windows rsync port requires Cygwin. Can't vouch for how well it actually works since I've never tried it. Maybe it's because I'm not that wild about loading Cygwin on a server. (S'truth I'm not too wild about Cygwin in general, but that's probably just me.)

cwRsync is a yet another packaging of Rsync and Cygwin. You can use cwRsync for fast remote file backup and synchronization. Rsync uses the Rsync algorithm which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand. At first glance this may seem impossible because the calculation of diffs between two files normally requires local access to both files.
Rsync normally uses ssh for communication. It requires no special privileges for installation. You must, however, have a working ssh system.

Alternatively, rsync can run in `daemon' mode, listening on a socket. This is generally used for public file distribution, although authentication and access control are available. Cygwin is a Linux-like environment for Windows. It consists of a DLL (cygwin1.dll), which emulates substantial Linux API functionality, and a collection of tools.

Installation

Supported platforms: Client - NT/2000/XP/2003/2008/Vista/7, Server - NT/2000/XP/2003/2008/Vista/7.

If anybody tries it please let me know?

[40hz]

@4wd - do you know if DataSafe also backs up servers: or just workstations from the server?

[4wd]

@4wd - do you know if DataSafe also backs up servers: or just workstations from the server?

-40hz (January 07, 2011, 07:55 AM)

Do you mean does it work like rsync, where you need a client/server relationship?

Then no, it's more like your traditional Windows backup program, (ala SyncBack, etc), choose a source and destination, (which can be FTP, CD or Network Share), choose your options, (Full, Incremental, Differential, Mirror; Compression, Encryption, Versions, etc) and hit go.

If you choose a backup type of Mirror, then Hardlinks are enabled.

I haven't extensively tested it, (I've got a few small OS gremlins that I need a re-install to fix, eg. VSS isn't working), but it seemed to work OK.

eg.
1st Run - 472 jpgs/pngs (1.78GB) copied in 32s (HDD -> HDD)
2nd Run - no changes, 472 files copied in 4.5s via hardlinks  (2 or 3 seconds is because it tries to get VSS working)
3rd Run - Added another 51 jpgs, 523 files copied in 6.9s, 472 via hardlinks

Each backup run goes into it's own sub-directory named Jobname.backuptype.runnumber, (eg. Test.Mirror.00008), and then the full directory structure of what was backed up.

I haven't tested it to my FreeNAS shares, (UFS), might try that tomorrow, (technically already tomorrow here but it's 0433 ).

Update: Works fine to FreeNAS shares on UFS drives, happily creates hardlinks for unchanged files which reduces space requirements a fair bit.

It's quite amazing, (well to me who hasn't had experience with links), seeing six (6) backups of the exact same files and not see the available disk space drop.  Effectively, 2GB of space used to store 10GB of uncompressed data - very neat.


I think I've just created a conflict in my brain....I like the speed/versioning of DataSafe but I have SyncBack Pro

Must...resolve...conflict........aaarrgghhhh!  <- Brain implosion.

There is something called cwRsync that will allow you to easily integrate the rsync daemon with various flavors of Windows. Catch is the Windows rsync port requires Cygwin.

-40hz (January 07, 2011, 07:50 AM)

There's also DeltaCopy.

In general terms, DeltaCopy is an open source, fast incremental backup program. Let's say you have to backup one file that is 500 MB every night. A normal file copy would copy the entire file even if a few bytes have changed. DeltaCopy, on the other hand, would only copy the part of file that has actually been modified. This reduces the data transfer to just a small fraction of 500 MB saving time and network bandwidth.

In technical terms, DeltaCopy is a "Windows Friendly" wrapper around the Rsync program, currently maintained by Wayne Davison. "rsync" is primarily designed for Unix/Linux/BSD systems. Although ports are available for Windows, they typically require downloading Cygwin libraries and manual configuration.

DeltaCopy doesn't require CygWin, all Client/Server binaries are in the one installation binary and it's very easy to set up.