Not backing up will cost you!

[JavaJones]

So here's part 1 of 2 blog posts I'm writing about my data loss, recovery, and backup experiences. I hope it will serve as a stern reminder to everyone to backup their important data. Not doing so could cost you 1000's of dollars or worse.

Now, I felt pretty secure with the data on the 4big given it was RAID5. For those not familiar with the technology, it basically uses multiple disks with a sophisticated data distribution system that allows for redundancy. This means that theoretically an entire disk can fail and your data is still ok because it can be rebuilt from the other disks. If 2 disks fail simultaneously (or the controller fails), then you have a problem. Theoretically however the chance of a double disk failure is lower than that of a single failure, so one would imagine the data is safer than with a single drive.

Unfortunately double disk failure or single disk failure combined with other corruption can and does happen, as I found out much to my dismay. I loaded all my photos onto the unit shortly after my return and began sorting through them and posting new sets every day or two. After a week or so of working on photos off and on, I started to see some issues reading certain images. I checked my Windows event log and found a whole bunch of disk-related errors essentially saying my 4big drive was corrupted and it needed to be scanned for errors. I rebooted shortly after and a disk scan ran automatically. Though I've never had much faith in Windows' chkdsk utility, I soon found out that it's even worse to run it on a RAID.

http://oshyan.blogspot.com/2010/12/seriously-people-back-up-your-data.html

- Oshyan

[cyberdiva]

Thanks, Oshyan.  It's a very compelling and scary account.  And thanks to you, I've just finished backing up my desktop.

[JavaJones]

Hooray! I've done some good.

Thanks for reading.

- Oshyan

[AndyM]

...Now, I felt pretty secure with the data on the 4big given it was RAID5. ...

But what about a house fire or burglary?

[JavaJones]

Absolutely, those are two good examples of unlikely but very possible scenarios that on-site backup (even in addition to RAID) will not cover you for. In fact, although I didn't write it in that particular blog entry, I had been planning to create a full backup system long-term. I just didn't worry about it *as much* or feel as urgent about it because of the RAID5 setup, and that was a mistake (and no burglary or fire was needed to prove that to be true).

The 2nd post will be up in a few days and will talk about the need for off-site backup and how to arrange it.

- Oshyan

[worstje]

I'm glad I didn't go through that.

I must say though.. I have a similar experience which caused me to drop RAID as a whole.

RAID is meant to to improve reliability, at least in the variety you mentioned as well as the variety I used. But in practice, one drivers/controllers RAID is a totally different beast from the next. So once stuff reaches something that can't be automatically handled (which is surprisingly often the case), you're up shits creek.

Why does it happen so often? Because you tend to set up an array using the same hard drives. Often, if you order them at the same time, it is the same batch. So once one drive breaks, the other one isn't far away. That is what happened to me.

Simply using non-RAID, and setting up a proper backup plan has proven to be a far more resilient course of action for me. Yes, once one drive breaks, I am stuck a few hours as I fix my situation up to get stuff back out of my backups. But my backup drives do not spin 24/7 like my main drives do, and drives failing is still supposed to be a very infrequent happening. So, once shit does happen to hit the fan, I find that I am in a far more comfortable situation regarding fixing stuff myself.

Seriously, do the math: buy a few more harddrives, which costs, say, $200. Or end up going your route, and shell out $4000 once stuff really goes wrong. And with my method, I have the flexibility to have my backups go to an external site (less chance of theft/fire/etc beating my backups through conventional means).

And that is why, for me, RAID is an expensive concept consumers tend to not really need.

Edit: seems someone else beat me to the 'other accidents' point of view.

[JavaJones]

Yep, you are exactly on target. And even though I got a refurbished 4big unit back after this (since DriveSavers can work with Lacie support for in-warranty issues), I think I am ultimately going to migrate away from it. It's just too problematic (for example I have a hard time getting this unit to work reliably with eSATA, and with USB 2.0 it's slower than an internal single drive so not much point).

- Oshyan

[Renegade]

Patiently awaiting part 2.

I had a server at LayeredTech with Raid 1 mirrored drives and when 1 drive died, they managed to lose the data on the other... Morons... Nothing to be done about it. Thankfully we had backups.

I've been meaning to get an automated backup solution going for important data, so I'm eager to hear what you've got to say on the topic.

[superboyac]

I'm revising my backup system next year, 2011.  I need some help in the best way a home user can add several disks.  I currently have 6 hard drives all connected to my desktop in various ways.  I want to have a more commercial type of solution.  I want to build a server, possible a Windows Home Server, that will house 10 drives or more.  I need some kind of rack or special tower that can hold around 10 disks (give or take).  I'd like to be able to slide disks in and out easily.  I don't know what is out there that can do that.  It's not something easily found in the consumer stores, and I'm not familiar enough with servers and such to have an idea about the good brands, models, or exactly what the words are for the technical terms.  But I want a separate piece of hardware that can house about 10 disk drives and I'd like to connect it through esata or lan to a server computer, I have no idea how to do that connection.  I'm guessing it's esata or an ethernet cable, but I don't know.  I am NOT looking for a NAS, as in one that is self-sustaining and has it's own OS on it and everything.  I just want the thing that holds the drives.  So please teach me about all of this if you know.  Thanks.

[4wd]

I just want the thing that holds the drives.

Addonics

RAID Tower IX - 15 hot-pluggable SATA drives, connect via eSATA, USB3, USB2 or Multilane^*  $729->$779 depending on whether you want Multilane or not
RAID Rack - 20 hot-pluggable SATA drives, connect via eSATA, USB3, USB2 or Multilane  $1219->$1619 depending on whether you want redundant PSU or not.

They use hardware RAID controllers, the RAID mode is easily set by turning a switch at the back to the mode you want.  You can also keep them as individual drives but if connecting by eSATA your motherboard chipset needs to support Port Multipliers, IIRC, any Intel chipset ICH10 or later.

I'd love to buy some of their stuff but it's so ridiculously expensive over here.

^* Multilane is 4 SATA channels configured into one Infiniband connector to provide maximum throughput, ie. 4 x 3Gb/s SATA channels instead of a single 3Gb/s eSATA channel.

Addendum: Otherwise for something a little cheaper to start with and more portable, there's their RAID Tower IV or their Mini Storage Towers.

[JavaJones]

Hmm, why do you need 10 drives? And why swappable? I'm not invalidating your needs, but I always feel like it's less productive to start with a partial solution and try to make that happen, rather than starting with basic needs and trying to address those in the best way possible. So what is your *most basic need*, the root of why you feel you need 10 drives and swappability?

For my part, for example, I have 6 internal SATA 2TB drives in my media machine. I have therefore 12TB of internal storage (it's not handled as a RAID). And it's hard to fill that up unless you're a major media hound like I am. 10 drives just seems insane when 2TB drives are available so inexpensively, it's hard to imagine needing 20TB.

Also, do you have an existing system that you would want to attach your storage to, or would you want to build a new system anyway to act as a "server"? If the latter, then although 4wd's suggested external array cases are good, I might just recommend a full tower case for your system build, something server-oriented can get you 10 or more externally available 5.25" bays. Then add some swappable enclosures and you're set (some even come with swappable enclosure hardware). Cost might be similar, and the external unit has the advantage of portability if desired, but it's nice to have something all-in-one sometimes. Here's an example server case with 8 hot swap bays: http://www.newegg.com/Product/Product.aspx?Item=N82E16811192058
or a nice Lian Li with 12 external 5.25" bays:
http://www.newegg.com/Product/Product.aspx?Item=N82E16811112159

- Oshyan

[Eóin]

Often, if you order them at the same time, it is the same batch. So once one drive breaks, the other one isn't far away. That is what happened to me.

I've done that myself so I'm no saint, but doing do is very much wrong. Proper RAID arrays should contain drives from as many different sources as possible.

[JavaJones]

The complexity of creating a "proper" RAID array is just one more reason it's not really a very good consumer-level option IMHO. Are there any retailers out there that offer hard drives in RAID bundles, say 4 drives all selected from different batches? Sort of like how multi-channel RAM is sold. Retailers would seem to be in the best position to make such a thing easy. Otherwise you have to find a good price for drives from 3 or more sources.

- Oshyan

[Dormouse]

I have to agree with all the concerns about using RAID; reflects my own thinking. I still have a NAS set up with RAID. I'm not especially anxious about it as I have everything duplicated elsewhere (& not had any problems with it so far) but redoing it without RAID is on my list of things to do. I feel that multiple copies, in multiple places is the best way to be sure you don't lose all access to your data.

[superboyac]

JJ,
I guess I don't need them to be swappable necessarily.  I just love the cases where you can slide the drives in and out without screws.  Or they are on a rack that you can pull out and deal with just the screwing of the drive in the rack.  My current case is like that.  What I'm saying is that I don't want a box where i have to stick the screwdriver in some funky angles inside.  I hate that.

That Lian-Li one is kind of nice!  That would be fine I think.  I can have all the drives mounted using the bays and whatever enclosure I want, and the motherboard and everything for the server is inside also.  Are there any other models or products like this?  I'd like to compare a few.

[superboyac]

4wd, those Addonics products are very interesting also.  A little higher end, but hopefully good quality and worth it.  I do like the idea of keeping it modular.  I'm normally very inclined towards modular solutions.  Computer housed one box, and drives in another.  Hmmm.

[superboyac]

Anyone familiar with Sans-Digital?  I'm eyeing this one:
http://store.sansdigital-shop.com/totr8baysass.html

It looks to be better made than Addonics.  I'd like to have as much metal as possible, minimal plastic.  But looks can be deceptive of course.

[superboyac]

Aha!!  I think I just found the most awesome thread for figuring this out:
http://hardforum.com/showthread.php?t=1393939

I'm sure to find something I like in there.

[superboyac]

I think I may have found my solution:
http://www.norcotek.com/item_detail.php?categoryid=1&modelno=RPC-4020#

I like this company, I think.  They seem to have good customer support and their stuff is relatively cheap vs. the other guys.  I'm building me a JBOD server, baby!!

[Renegade]

I think I may have found my solution:
http://www.norcotek.com/item_detail.php?categoryid=1&modelno=RPC-4020#

I like this company, I think.  They seem to have good customer support and their stuff is relatively cheap vs. the other guys.  I'm building me a JBOD server, baby!!

I had a hard time finding the power requirements, but eventually located them:

Power Requirements: Expect brown-outs in your neighborhood

[JavaJones]

You really are a bit nuts I think! In my eyes those are all way overkill solutions for most people's needs. Do you even have a rack to mount the NORCO unit in? You haven't given any details on exactly how you intend to use all that space and why you need it in the first place, so it's hard to say what's appropriate and necessary. Again not to say that you *don't* need it, but knowing how you intend to use it will help recommend the best solutions.

For example if your biggest reason for wanting to use RAID is so you can have one giant storage unit with all your stuff on it, consider the option of something like Win7's "library" system. Even if you have e.g. 10 separate drives, you can have all your movies, music, or whatever showing up in a single library, pulling content from multiple independent hard drives.

So far it seems like you just want to play with slick hardware more than anything though. Of course there's nothing wrong with that if you've got the money to back it up.

- Oshyan

[superboyac]

lol, it's really not that crazy.  As it is I have almost 4 TB of data that is backed up, some of the data has double redunancy backup.  So, just right now with my simple desktop, I need 8-10 TB worth of hard drives.  Then I have all sorts of things I haven't even put on the computer because I'm running out of space.  In the coming years, I plan on digitizing everything, and when you start talking about video and audio, uncompressed and raw, we're talking big stuff here.  So just keep in mind that I haven't even really STARTED doing that yet.  THen, there's also the issue of some business stuff that I'm working on, and backing up not just my data, but business data, and other friends and family's data, and it will get big real quick.

I don't want to fiddle around with all these enclosures and external drives I have right now.  I have a few drives in my tower, i have a couple attached externally, I have a couple just lying around, and a couple in use in other computers, because there's no central server.  I want to consolidate it all.

It's not that expensive either.  That Norco box only costs $300.  Then all I need is the serving computer.  So I think i will build a small or mid size rack no more than 3-5 feet high, and put it somewhere.  I'm also thinking about putting a LAN in the house so each room can connect to the server hard-wired.  But everyone says that wifi 802.11n speeds should be plenty fast enough for me, but I have yet to see it to believe it.  Can I stream 1080p over a wifi connection in the house?  I doubt it.

[4wd]

I'm building me a JBOD server, baby!!

Geezz, the perfect way to lose everything through the failure of one drive.

[superboyac]

I'm building me a JBOD server, baby!!

Geezz, the perfect way to lose everything through the failure of one drive.

Wait...huh?  I don't understand why.  I'm not raiding.  i'm just putting a bunch of disks in a rack.  Each disk will have another identical disk (or two) for backup up (not images, but file syncing).  How would I lose everything?  If what you say is true, then even my current setup is at risk.  but I don't think what you are saying is true.

[4wd]

I'm building me a JBOD server, baby!!

Geezz, the perfect way to lose everything through the failure of one drive.

Wait...huh?  I don't understand why.  I'm not raiding.  i'm just putting a bunch of disks in a rack.  Each disk will have another identical disk (or two) for backup up (not images, but file syncing).  How would I lose everything?  If what you say is true, then even my current setup is at risk.  but I don't think what you are saying is true.

I think that maybe I may have misunderstood because of your use of the term JBOD.

Generally, JBOD refers to combining separate HDDs into effectively one bigger HDD so that data is spanned across all of them.  So if the 3rd HDD of a 5 HDD array dies, you've generally lost all data across all HDDs.  You may be able to recover files using recovery software from the other HDDs but it'll be a long process.

By definition, JBOD is non-RAID and therefore there is no data redundancy.

Your explanation of what you're going to end up with is kind of a RAID-1 setup for each individual HDD without actually using RAID.