DonationCoder.com Forum
Main Area and Open Discussion => General Software Discussion => Topic started by: superboyac on January 04, 2011, 10:25 AM
-
I'm curious because I will build one this year. I'm leaning towards Windows Server. But I know a lot of people will say that is overkill. I don't know. WHS sounds good, but I don't need any of the built in features because I'm a 3rd party software kind of person anyway.
Part of my question is, what does a server OS do that a regular OS doesn't do (like Windows 7, XP)? i mean,r eally, what's the difference? I'm guessing it has to do with handling a lot of drives, right? I know my XP machine starting acting weird once I attached more than 5 drives to it.
So that's what all this is about. I'm trying to choose a server OS for home.
-
The difference between a workstation and a server is the allocation of resources. In a workstation, more priority is given to running applications while on a server more is given to background services. Non-essential addons (themes, various client services, etc) are cut out in a server environment and a focus is placed on availability versus usability.
I use Server 2008 R2 for most of my work. I do have a dedicated server setup running pfSense (http://www.pfsense.org/) and another experimental one running centos and testing out apache on with a few specialized configurations for certain projects I am working on. Other than that, I rely on WS2008R2 Core (for the most part).
-
Linux - because it's gratis (I used to have a lot of spare time).
-
... what does a server OS do that a regular OS doesn't do (like Windows 7, XP)? ...
-superboyac
The difference between a workstation and a server is the allocation of resources. ...
-Josh
Is this a function of the hardware or the OS?
-
The OS allocates resources as demanded by users or applications.
-
Windows Server 2008 R2 because it's free for students and is just sooooo easy to manage.
If I were setting up a web facing server I'd prob go Linux or *BSD. Actually I'd almost certainly go OpenBSD for geek browny points :D
-
Windows Server 2008 R2 because it's free for students and is just sooooo easy to manage.
If I were setting up a web facing server I'd prob go Linux or *BSD. Actually I'd almost certainly go OpenBSD for geek browny points :D
-Eóin
Thanks.
-
BTW, one big plus for Windows Server over a desktop Win OS is that it supports RAID setups. Pretty much all ours does at home is act as a file server tbh.
-
Windows Server 2008 R2 because it's free for students and is just sooooo easy to manage.-Eóin
I'm MCSA in both Win2k & 2k3, and yet I still spend half the time working with 2k8 trying to figure out where the hell they moved stuff to. I think they got a little too cute with the interface; the new IIS really drives me nutz. The old one was fine (UI wise) what was the point of scattering stuff everywhere hidden behind little pictures?!?
I'll get used to it eventually, and really like parts of it ... But I'll bitch about the rest in the interim... ;)
-
Well I only started playing with Win Server OSes with Server 2008, so i guess I started from a clean slate.
-
Well I only started playing with Win Server OSes with Server 2008, so i guess I started from a clean slate.
-Eóin
That can help I'd imagine, starting with 2k I didn't have to unlearn any of the NT4 stuff so it was easy to pick-up as everything was new. The term Primary Domain Controller (or PDC) is a left over from the NT4 days - You still hear people using it, but they're generally the (older) NT4 crowd - still not completely unlearned yet... :)
...and I guess I'm getting my turn at that bat now also... (hehe) ...Shit.
-
The main differences between server and desktop OSs as I see them are:
- Built-in services - things like IIS, Active Directory, etc. (for the Windows side)
- Resource allocation/priority (as Josh said)
- Different default services - servers will often not include services (or have them disabled by default) that desktop systems use, and have other services enabled/installed that are not available or enabled by default on desktops
- More sophisticated user handling and credential management
- More sophisticated data storage management (e.g. RAID)
There are of course a few other things, but those are the majors I think.
Now the first question that springs to mind is, if you're going to be using all 3rd party services, what exactly do you want a "server OS for"? Especially Windows Server. Even for a small business, if you have less than 10 clients (at a very, very minimum) I would not recommend a Windows Server setup unless there were serious authentication/credential management and service needs.
If this is just for your home network environment, then what is driving this decision? Do you simply want a central data repository where everyone can access e.g. movies and other media? Is centralized backup a desire? Do you want to run your own mail server so that the whole family has on-site mail with central SPAM/virus filtering, unlimited mailbox size, shared calendars, etc? Essentially, as always, what problems are you trying to solve?
Personally if I had a desire to create a server (I don't, despite having 6+ systems at home and more than 20TB of data to access in a shared manner), I would probably use Linux, even though I'm not terribly familiar with it, just because I know it can cover all the services I would want without cost. If I didn't want to host my own mail or web services, I'd go with Windows Home Server. Windows Server 2008 and other business-oriented versions wouldn't really enter my mind because I have no need for Active Directory and such services, and if I want to run a web server, I want it to be LAMP anyway, not IIS.
- Oshyan
-
OK, for the sake of this discussion, let's avoid any concerns about price. Let's just assume I have an infinite amount of money and it doesn't matter what the alternatives cost. I may go with a very expensive solution simply because it means I have more buttons to pick instead of command line stuff. I'm like that. I'm VERY familiar with Windows, and while I'm not necessarily afraid of Linux (I've used it as a virtual machine), if it means I need to fiddle around more with command line stuff or text files, I'd rather not.
JJ, your assumption about my needs are pretty right on. Definitely less than 10 computers will be using the server. My main purpose is simply file storage. I want a central place to put a bunch of hard drives and access them from several machines. So, I'm talking about anywhere between 5-20 drives, and 10-20 TB of information, including double and sometimes triple redundancy. So I don't actually have that much data, but once you multiply it for the backups, I do have a lot.
The backup stuff will be handles by SFFS. These are all just file syncing, nothing more. No RAID. I want to be able to pull a drive out at anytime and use that drive's contents standalone anywhere else...so just files and folders. Some of the drives will be used to stream media, so the drives that have things like videos will be used by other computers to play them. No big deal, I think. Nothing normal OS's don't do already.
I mean, what if I just ran regular ol Windows 7 on the server machine? Would that work? Is it going to have issues with all the hard drives? I'm not going to us Linux if I have to get used to a lot of new things like a new type of file structure, command line things, different programs to learn. i have no desire for that. I'm a Windows guy, I'll stick with it until I have time to explore other options. I'm extremely good with Windows and very comfortable getting all up into it without worrying about it.
I'm not that concerned about crazy security things, like corporate level firewalls and so forth (unless I should be). I mean, I'll do it if it's relatively painless. But i don't want to buy thousands of dollars of hardware firewall equipment, and learn all the security things. I've heard that just my normal wifi router is good enough, no? I mean, I recently tried setting up an SFTP server, and I never got it working. It was a huge pain. I had no idea what the hell I was doing. I don't have the time now or later this year to learn too many new things. but I do need to do something about my storage issues, because I am full right now and I'm tired of burning DVD's and keeping track of what I'm keeping/removing/backing up. I literally spend 2-4 hours a week now just dealing with running out of space issues.
Anyway, if there is a nice Linux distro that handles this well and wouldn't require me to spend ages on Linux forums, I'd be very interested in checking it out.
-
If the vast majority of your need is simply for data sharing and syncing, I believe a NAS-oriented solution (like FreeNAS) might be ideal. That being said, if you're most comfortable with Windows, there's no reason not to just use Windows 7 for this. It can handle the number of drives just fine (I've had up to 15 drives on a Win7 machine already), it has Library support, you can share Libraries directly, meaning you can essentially "federate" data across multiple drives into a single cohesive presentation and then share it. Being Windows, you can use all the tools you're already familiar with. It won't include extraneous web server, mail server, and active directory type stuff, which are all services you won't (as far a I understand) need and will just be there for no reason. Regular Windows Homegroup sharing should provide sufficient access control and ease of use, especially vs. e.g. Active Directory.
Security should not be an issue as long as you're using appropriate encryption on your wireless (WPA2), and you do not share files with "Everyone" but rather have user names, passwords, and appropriate logins and permissions on each machine for the central share(s). This is fairly easy to setup, and will provide another layer of security should a casual snooper happen to get in to your network (say a friend who comes over who you give the wireless key to).
All in all I think Windows Server is overkill and you should be looking at either a NAS-in-a-box type prebuilt Linux distro, or Windows 7. Probably the latter given your inexperience and level of comfort with Linux, plus the fact that you want to run your own familiar sync software.
- Oshyan
-
Thanks JJ, that's exactly what I was looking for. If what you say is true, I'm very happy with that setup and would not be nervous about diving into it right now.
-
Well, I speak from personal experience. This is exactly what I do with my media server. I actually run 2 "servers", one for media which doubles as a media playback and cataloging machine (it's in my living room hooked to my flat screen TV), the other is a more general data storage system that handles other (non-"media") data sharing, backup/syncing for all the other systems, cloud backup (it backs up all the other systems to its own drives, then backs *those* up to a cloud service), etc.
Anyway, Win7 on both machines, works great. No RAID for the internal drives, Libraries functionality is ideal for the media folders. I use an off-board RAID unit at the moment (the Lacie mentioned in my "Not backing up will cost you" thread) on the non-media server, but probably not for long given my experiences with it.
- Oshyan
-
Let me make this simple... Technology Totally Sucks ... When Applied Badly.
Windows 7 Libraries Do Not Work over the wire. Period. So if you include network share folders X, Y, and Z in a Library on one machine, then go to a different machine you will have to recreate in again, and again, and again ... There is a word for that, it's called stupid.
A Windows Server... Will give you the full set of file sharing services, like DFS (Distributed File System) which is a "Library" for folders from disparate locations pulled into a single shared location for consistent easy access from any location...And you only have to do it once. No you don't get the spiffy indexing features with DFS, but you also don't get them with the Win7 Libraries over-the-wire either. So nothing is "lost" but major constant headaches.
IIS, AD, DHCP, DNS, and etc. are all services that Can Be Added to a Win2k3 server. They are not compulsory, or even there by default. It does just fine as a bog stock workgroup server. However IIS is a kick-ass web server if you know how to setup and manage it properly.
With 10TB of data to keep track off if you go with proper commercial hardware, then RAID (1+0 as well as a backup strategy) would truly be the best configuration. But if you just want to hang a bunch of independent drives off a RAID controller then DFS will be your best bet to keep things organized and accessible. Either way a true server OS will handle the access of the data much better by not wasting it's resources on themes & multimedia nonsense.
Note: Commercial RAID controllers have on-board repair features that work (both manually and transparently) quite well. Not to mention that multiple configuration can be used for the various drives so you're not locked into the all or none crap of the consumer lever stuff.
-
Oh man...
OK, I think I'm going to have to set aside a couple of weeks in the next few months to explore all of this. I have no idea what is going. i don't know the differences in all of these things, i don't get it. I'm hearing all sorts of contradictory things, I'm not going to be able to make a decision this way. It's ok, I appreciate the help, but I think this is going to take a lot of legwork and time on my end until I know what I'm doing.
-
Libraries work fine for what I'm using them for, but I do not attempt to combine multiple *remote* network folders into one library, I'm going the opposite direction, taking multiple local folders and presenting them as a single share. It sounds like suberboyac wants to do the same thing.
What exactly is the advantage of RAID in this situation? If you're not using AD, IIS, etc. then what's the point of using a more expensive server OS? Not having skins enabled? Turn them off in Win7. Server-tuned process scheduling? Set priority in your Windows system settings to services. Yes, of course, these are not quite the same thing, but for 90% of the benefit they're worth not bothering with the expense and, yes, added complexity of Windows Server IMO.
- Oshyan
-
SBAC, The big question is what is the ultimate purpose of the machine in question? That can help us tailor an OS/Solution to your needs :)
-
Oh man...
OK, I think I'm going to have to set aside a couple of weeks in the next few months to explore all of this. I have no idea what is going. i don't know the differences in all of these things, i don't get it. I'm hearing all sorts of contradictory things, I'm not going to be able to make a decision this way. It's ok, I appreciate the help, but I think this is going to take a lot of legwork and time on my end until I know what I'm doing.
-superboyac
You know Windows 7 already. You can try this stuff on smaller data sets (e.g. shared libraries) and see if it does what you want. If not, go for the bigger guns. If so however, you've saved yourself time and money.
It's really all about accurately and completely (as much as possible) determining your needs and then mapping the solutions that exist to those needs as closely as possible. If for example you have a bunch of folders on separate machines that you want to access as a single index of files, and you want every machine on your network to be able to do this, then Stoic's suggestion makes more sense than Win7 default. But I don't need to do this, and it didn't sound like you do either. Instead I have 4 network shared libraries - music, movies, tv, and data, the first 3 of which are on the media server, the latter on the data server - and these are linked to on all my machines here. It's a pretty easy and convenient setup as far as I'm concerned.
- Oshyan
-
In my experience the communication between any PC that runs XP/2003 (or older) and the one that is running 'HomeGroup' (7, 2008R2) is not great...to put it mildly. If you plan to use such a mixed environment, than I strongly suggest to not use the 'HomeGroup' crap at all.
In Windows 7 it is possible to set up your networking for a 'Work' environment, which appears to "kill" the HomeGroup features.
According to Fred Langa (http://windowssecrets.com/comp/101014) you can actually gain 10 to 15% of raw networking speed just by disabling 'HomeGroup'.
-
Interesting tidbit Shades. I haven't had any major problems with the few XP machines I have left, but admittedly they seldom access the network resources. Everything else is on Win7. The network speed issue is of greater interest to me though...
Edit: To clarify, I'm not actually using "Homegroup" in my setup, though I'm not sure I've properly and fully disabled it. But I did suggest that above just because it's pretty easy to get working as long as you have other Win7 machines. But superboyac is surely savvy enough to use regular Windows networking and given the advantages you've mentioned, I think it's worth doing that, especially in a mixed environment.
- Oshyan
-
@SB-
A server's primary function is to provide security. It does this by controlling access to resources and data assets stored on a network. Anything else a server does after that is pure gravy.
Some suggestions in no specific order:
- If you don't need much security - and all you want to do is store and share files - a NAS solution is your best bet.
- If you want/need to do more than that (i.e. provide remote access, have user roles, offer additional services such as VMs, HTTP or SFTP) then you will want to get a 'real' server.
-For home or SOHO use, Windows Home Server is all you'll need. It's very easy to work with. And it doesn't take a pilot's license to fly it. At a street price of around $100 (so far) it's also pretty cheap for a product that has full tech support available.
-Don't even bother running print shares off a home or SOHO server. Unless you need to restrict who gets to use the printer (or account for the number of pages coming off it) just go for a printer with network capabilities. Print directly to it over your network and be done with it.
For a business with plans to grow (or just delusions of grandeur) the choices get a little more complicated.
-If you're going to maintain it yourself, it doesn't really matter what you pick. Linux or Windows - either way you've got some work and book time ahead of you.
-If you're hiring, pick whatever the most popular platform in your area is . Because that's what the talent pool you're going to hire from is likely working with - and knows best.
In my neck of the woods, it's Windows Server hands down. Out in sunny Los Angeles or Frisco Bay it's probably more likely to be an even split between BSD and Bill Gates.
-If you want to try a general Linux server solution, and it's your maiden voyage, try one of these first: Zentyal (http://www.zentyal.org/) or ClearOS (http://www.clearfoundation.com/). They're very forgiving since they have a nice GUI to work with until earn your Techno-Wonk Beanie-copter.
[ You are not allowed to view attachments ]
(Note: Effective 10/17/1999 - having webbed-feet is no longer a requirement in order to wear a 'B-C.')
Luck! :Thmbsup:
-
Well said (as usual) 40hz. Security is of course one biggie I missed in my list above, though several of the entries sort of tie back in to that. Basically, access control.
I haven't worked with WHS myself, but I'm curious about it. Like I said above, Win7 (Home Premium in case anyone is curious) works fine for my needs, which I think are pretty extensive for a home user. But WHS might fulfill my needs even better. I'm just a bit wary after they removed the storage concatenation feature recently, and I'm unsure what the future of the product is.
Also a good recommendation as far as printing goes. My wireless all-in-one Canon has been awesome in this regard. Seamless multi-user access.
- Oshyan
-
No vote there as I don't need a server for home use. Regular desktop OSes are just fine.
I work at home, and use servers for work, but that's another matter. For that, it all depends on the work I need to do.
-
@SB-
A server's primary function is to provide security. It does this by controlling access to resources and data assets stored on a network. Anything else a server does after that is pure gravy.
Some suggestions in no specific order:
- If you don't need much security - and all you want to do is store and share files - a NAS solution is your best bet.
- If you want/need to do more than that (i.e. provide remote access, have user roles, offer additional services such as VMs, HTTP or SFTP) then you will want to get a 'real' server.
-For home or SOHO use, Windows Home Server is all you'll need. It's very easy to work with. And it doesn't take a pilot's license to fly it. At a street price of around $100 (so far) it's also pretty cheap for a product that has full tech support available.
-Don't even bother running print shares off a home or SOHO server. Unless you need to restrict who gets to use the printer (or account for the number of pages coming off it) just go for a printer with network capabilities. Print directly to it over your network and be done with it.
For a business with plans to grow (or just delusions of grandeur) the choices get a little more complicated.
-If you're going to maintain it yourself, it doesn't really matter what you pick. Linux or Windows - either way you've got some work and book time ahead of you.
-If you're hiring, pick whatever the most popular platform in your area is . Because that's what the talent pool you're going to hire from is likely working with - and knows best.
In my neck of the woods, it's Windows Server hands down. Out in sunny Los Angeles or Frisco Bay it's probably more likely to be an even split between BSD and Bill Gates.
-If you want to try a general Linux server solution, and it's your maiden voyage, try one of these first: Zentyal (http://www.zentyal.org/) or ClearOS (http://www.clearfoundation.com/). They're very forgiving since they have a nice GUI to work with until earn your Techno-Wonk Beanie-copter.
(see attachment in previous post (https://www.donationcoder.com/forum/index.php?topic=25115.msg230088#msg230088))
(Note: Effective 10/17/1999 - having webbed-feet is no longer a requirement in order to wear a 'B-C.')
Luck! :Thmbsup:
-40hz
nice! This goes in my notebook.
Based on this, I know what I will do. I'm going to start with Windows Home Server. If my needs grow beyond that, i will deal with it then. 5 years ago, I would have gone with the most hardcore solution, probably Windows Server in this case. But I'm different now, I've learned my lessons. Very glad to hear, and I know most people here have said very positive things about WHS.
-
I don't have the kind of extensive experience and knowledge demonstrated by others here, but I've been using a Linux server for several years (from Redhat to Mandrake to Fedora to Ubuntu currently). I have never tried Win Server so I can't compare, but I'm very happy because:
1. it's rock solid; never crashes unless the hardware fails (see the next point).
2. it works on very old hardware, so when the server dies, I get to buy myself a new desktop, move my old desktop to my wife's desktop, and then move hers to server. This happens every few years, very good excuse for buying new hardware and trying a new distro. :)
With a good backup practice, switching servers is easy.
3. No need to pay big bucks for server software. I learned Postfix, Spamassassin, Dovecot (for email) and Samba (for Windows file sharing) first, then BIND (DNS) and Apache (web), and then gradually onto other stuff. All standard stuff on *nix land, with plenty of reference material and easy to get questions answered.
4. Easy to manage remotely. My server sits next to my desktop on the floor, but it has no monitor, no keyboard/mouse attached to it. All I need is Putty (except when setting up a new system of course). I edit all the config files with my favorite editor on Windows; no time (and no need) to learn EMACS or VI.
-
Server 2008 R2 for me. I have WHS running on my file sharing box but with DE being dropped and newer 4k sector drives not being supported, I've decided to move away from it. I looked at various alternatives such as FreeNas, OpenFiler, even ZFS but I simply don't know Linux well enough and most open source software runs great on Windows anyway.
I will be installing Server 2008 R2, running WHS for pc backups in a VM, and maybe a few Linux distros in VM's as well. I look forward to trying out some specialized server software as well.
-
SHIT I just spent a freaking hour typing a response and the damn thing just vaporized. Board ate it or something, hell I don't know.
Now I gotta do the whole damn thing over - Where the hell is Ctrl+S when you need it! :(
-
Let me just start by saying that I have seen way to many people lose thousands of dollars taking shortcuts with data storage, Okay...
What exactly is the advantage of RAID in this situation?-JavaJones
Redundancy. If you have one drive, and that drive blows ... I mean really blows with hard parts touching or seized (which is rather common) ... you are sunk.
Now I know you just had a bad experience with consumer grade RAID hardware but you're throwing the baby out with the bath water. RAID is not the enemy, the appliance that was handling it was...
(RAID5) Two drives blowing in perfect harmony is excruciatingly unlikely, but if you want the extra cushion there is RAID 1+0 which would require that you lose three drives simultaneously to lose anything ... now that type of configuration does requires commercial hardware, but...
If you are going to have 10+TB of data kicking around the house, it's time to get (serious) off the porch and get the type of hardware that was actually designed to store that much data in a safe and comfortable fashion. And by comfortable I mean be able to repeatedly spin-up that many drives to find the file(s) you just asked it for without frying the PSU because it got a tad dusty and couldn't quite handle the request without puffing a cloud of smoke.
Not to mention that the uber handy looking flavor-of-the-month NAS appliance isn't going to be repairable in a few years because there is no compelling reason for anybody to stock parts for it. It's an appliance ... like a toaster or a blender, they're not designed to be repaired. When they break...You throw them away. So if the proprietary RAID controller (or other parts) decide to go poof ... you ain't getting another one. But you are at the mercy of the data recovery companies, and I'd rather not think of what the bill for recovering that much data would be, would you?
Off-lease commercial hardware is cheap as hell, and being mainstream they are quite common and easy to find parts for even years later because there is a market for the parts. You get a real hardware RAID controller with built in diagnostics that actually work, and the ability to rebuild after a disk failure automatically on the fly (e.g. zero down time). Not to mention they are also insanely fast. Doing a full format on a 200GB partition takes about 17 seconds ... Try that with a NAS appliance.
Either way you are looking at a grand or so ... So why waste it on something that's unrepairable and designed to die if used as advertised. Sure the last generation commercial server won't look quite as sexy in the living room, but it'll fit just fine in a typical closet and run forever in a home environment under a scant fraction of the load it was designed for, for damn near the same price!
If you're not using AD, IIS, etc. then what's the point of using a more expensive server OS?-JavaJones
Who said it was more expensive?!? Off lease commercial hardware (which was already paid fore once...) will either come with or have available the necessary server OS license. It's perfectly legal and endorsed by MS. The link I provided in your other thread has a price tag of only $250 for a flat out MS certified completely legal copy of Windows Server 2003. Now how much is Window 7?? ;)
I've dealt with that company many times in the past, I'll be ordering a server from them tomorrow. They are for-real, completely legit, super fast, and the customer service there can only be compared to things I saw on TV shows from the 50s.
-
I still don't see a compelling argument for RAID here. You're basically just saying "pro-level RAID is more reliable". Yes, it probably is. And I guess if you buy it off-lease it's affordably inexpensive (I have no experience with doing that so I can't say). The question is whether it's significantly advantageous over basic software syncing *for the home user*.
We can't use the amount of data as a gauge of how "serious" or "important" anything is anymore. In times past only enterprises has 10+ TB of data, but these days with high resolution digital cameras, home DVRs, people ripping their own media, huge music collections, and everything else going digital, we're approaching a time where 10TB will not be unusual for the average household to have. Does that now mean *everyone* should have a RAID unit?
So again the question, what specifically are the advantages over basic syncing? It seems like the only real benefit you mentioned is speed. You could argue that realtime redundancy is good to have, and it is, but is it worth the added complexity vs. a simple sync solution? Think about what superboyac has said about the desirability of being able to just take a drive and use the data on it as-is. That's simplicity, for a home user.
And how much storage does that $1000 buy you anyway? Can you toss in 2TB drives to upgrade the storage? Does the controller support that? Are they SATA or SCSI? In other words does $1000 buy you a usable and *upgradeable* large data solution, or is it a one shot deal?
As for my own situation with the Lacie unit, it's getting a bit off-topic but your proposed solution isn't really applicable to me either. I dunno about you, but I don't equate a stand-alone unit I can just plug in to a USB/eSATA/Firewire port with a complete, separate machine with RAID. I guess I could just use that machine as a sort of NAS, but the whole point to me was to have *local data access speeds* over e.g. eSATA, rather than network-limited speeds.
- Oshyan
-
SHIT I just spent a freaking hour typing a response and the damn thing just vaporized. Board ate it or something, hell I don't know.
Now I gotta do the whole damn thing over - Where the hell is Ctrl+S when you need it! :(
-Stoic Joker
I know how aggravating that is. When doing long responses, I use Editplus, then just paste and post.
-
SHIT I just spent a freaking hour typing a response and the damn thing just vaporized. Board ate it or something, hell I don't know.
Now I gotta do the whole damn thing over - Where the hell is Ctrl+S when you need it! :(
-Stoic Joker
I know how aggravating that is. When doing long responses, I use Editplus, then just paste and post. -Renegade
It's all text (https://addons.mozilla.org/en-US/firefox/addon/4125/), mates :)
-
(RAID5) Two drives blowing in perfect harmony is excruciatingly unlikely-Stoic Joker
Yet it does happen - and when it does, it's usually when rebuilding the array, which seems to be more stressful on the disks than the simple procedure of re-duping a mirror to a blank drive. Shit's probably most likely to hit the fan if you've use drives from the same batch when building your raid5, which is a cardinal sin and all that... but with the stories I've heard, I wouldn't put my faith in Raid-5.
Limiting yourself to mirroring for RAID does mean you don't get partitions larger than a single drive, but I don't see that as a super big problem - decent filesystems can abstract that need away (NTFS junctions, unix symlinks, ZFS storage pools, ...).
IMHO the only real advantage hardware raid controllers give you is battery-backed cache. And replacements for those cards tend to be pretty expensive, don't they?
-
(RAID5) Two drives blowing in perfect harmony is excruciatingly unlikely-Stoic Joker
Yet it does happen - and when it does, it's usually when rebuilding the array, which seems to be more stressful on the disks than the simple procedure of re-duping a mirror to a blank drive. Shit's probably most likely to hit the fan if you've use drives from the same batch when building your raid5, which is a cardinal sin and all that... but with the stories I've heard, I wouldn't put my faith in Raid-5.-f0dder
I did say unlikely, not impossible, and with other configurations like RAID 0+1 you gotta blow all three at exactly the same time. Everybody has a horror story about an uncle that got blown to bits in a taco eating accident...The point is that it is highly unlikely.
Now a software RAID controller probably will torch your data if you look at it funny ... Which is why I like to avoid them.
Limiting yourself to mirroring for RAID does mean you don't get partitions larger than a single drive, but I don't see that as a super big problem - decent filesystems can abstract that need away (NTFS junctions, unix symlinks, ZFS storage pools, ...).-f0dder
Um... I don't recall making that distinction/recommendation, so I'm not sure what you're driving at here.
IMHO the only real advantage hardware raid controllers give you is battery-backed cache. And replacements for those cards tend to be pretty expensive, don't they?-f0dder
From the manufacturer? Staggering ... That's why you buy it from the other guy... :) The key is that the necessary part will actually be available.
The drive failure alarm is also a handy feature...especially if you have a headless server that doesn't allow you to see the first failed disk warning until it's too late.
Transplanting a RAID set is also smoother and safer with commercial hardware. When our PowerEdge 1600 started to die, I grabbed a PowerEdge 1800 for about $600 and since both used the same RAID controller (options, less is more) I was able to just drop in the second set and go. The controller pulled it in auto-magically, and Windows only needed an Import Foreign Disk reminder and we were live. Sure I could have restored the *)GB of data from tape - in a few days - But I like things simple. ;)
-
IMO, RAID is oversold when it comes to small server deployments.
FWIW I think it's generally a big mistake to try to do RAID on the cheap. Epecially if you're doing RAID-5. I've seen too many bad things happen with inexpensive RAID controllers and consumer grade SATA drives to be very comfortable recommending them.
I believe most people would be far better off with some combination of routine backups, smart folder synchronization, and restoration disk images unless they have the money (and need) to go with "server grade" drives and controller cards.
By all means go with multiple drives inside a small server. Just don't create arrays out of them unless you have a genuine need to do so.
----------------
(RAID5) Two drives blowing in perfect harmony is excruciatingly unlikely-Stoic Joker
Yet it does happen - and when it does, it's usually when rebuilding the array, which seems to be more stressful on the disks than the simple procedure of re-duping a mirror to a blank drive. Shit's probably most likely to hit the fan if you've use drives from the same batch when building your raid5, which is a cardinal sin and all that... but with the stories I've heard, I wouldn't put my faith in Raid-5.
-f0dder
+1 on that. I've seldom seen a rebuild go 100% smoothly. Usually all of the drives were purchased at the same time - and exposed to the same environmental conditions (i.e. heat) during their service period - so the likelihood of a second drive failing during a rebuild is not exactly uncommon.
In a mission critical setting, the only thing RAID really gives you is uninterrupted uptime and the opportunity to get the "most current" backup or image off the array. If the rest of the drives in the array are over a year or two old, you're better off replacing all of them.
If I got $5 for the number of times I've replaced one RAID drive element only to have a second one go within a few weeks...well...maybe I wouldn't be wealthy, but I'd still have a tidy sum tucked in my pocket.
IMHO the only real advantage hardware raid controllers give you is battery-backed cache.
Plus better reliability and (maybe) some speed gains along with reduced stress on the server. But on a modern server equipped with today's multicore CPUs and RAM configurations, I don't think it's all that big a benefit any more.
And replacements for those cards tend to be pretty expensive, don't they?
Less so than in the past. But they're still pretty pricey. :'(
-
One of the things I liked about Windows Server 2008, when hollow (https://www.donationcoder.com/forum/index.php?action=profile;u=34386) was running it, was that multiple users could be logged in at the same time, running whatever apps they needed to run, each having their own desktop.
Hollow had set up an account on his machine for me, which I accessed from my old slow WinME box over 33.6k dialup, using an RDP client. I was able to run stuff on his machine that I could never even dream about on my own box. Utilizing his broadband connection to download large PDF files and view them using his system's resources, it took me a fraction of the time it would have taken me to add the books to my ebook directory.
All while he was using his PC for whatever purposes he needed it for.
It got me thinking about how great it would be for family use, where you could give the kids an old junk PC with 9x and an account on your machine running Server 2008. Everything they would be doing would be through your machine, which you could keep the security locked down tight and easily control when they could and couldn't use it.
An old 9x box is dirt cheap to acquire (probably could even get one for free), and they are easy to maintain, especially when you don't have to worry about installing software on it, so if the kids damaged their pc in any way, resetting the OS would be quick and painless with no data loss (all their data would be on your machine). And if the hardware died, no sweat off your back to replace it with another junk PC.
-
I still don't see a compelling argument for RAID here. You're basically just saying "pro-level RAID is more reliable". Yes, it probably is. And I guess if you buy it off-lease it's affordably inexpensive (I have no experience with doing that so I can't say). The question is whether it's significantly advantageous over basic software syncing *for the home user*.-JavaJones
Well... with 10+TB of data ... Size Matters. Where are you going to sync it to?!? The cloud? Total recovery time = 9 weeks... *Joy* ...I ain't got that kinda time, do you? another NAS appliance? Ouch $$$.
We can't use the amount of data as a gauge of how "serious" or "important" anything is anymore. In times past only enterprises has 10+ TB of data, but these days with high resolution digital cameras, home DVRs, people ripping their own media, huge music collections, and everything else going digital, we're approaching a time where 10TB will not be unusual for the average household to have. Does that now mean *everyone* should have a RAID unit?-JavaJones
You gota back it up somehow, some where, and magnetic tape is too expensive for an archive that size even for most SMBs. Hell you're talking about a $10,000 device. Sure you can use one of those external drives (a stack of them actually) but do you want to walk a typical end user through an incremental restore? I don't.
So again the question, what specifically are the advantages over basic syncing?-JavaJones
Cost. Syncing requires either a 1 to 1 target size or you gotta decide what not to backup ... 'cause it won't fit.
It seems like the only real benefit you mentioned is speed. You could argue that realtime redundancy is good to have, and it is, but is it worth the added complexity vs. a simple sync solution? -JavaJones
How is it complex? You read the instructions, do it once, and then it continues working all by itself. Syncing requires that you remember to do it; or store you files in the right magic folder; or notice the error message... You aint gonna miss the alarm on a RAID controller going off ... Last time that happened here I had half the company in my office frantically pointing inside of 3 minutes. The other half of the staff...? ...Was outside thinking it was a firedrill.
Think about what superboyac has said about the desirability of being able to just take a drive and use the data on it as-is. That's simplicity, for a home user.-JavaJones
I did, parts of it are possible, and parts are science fiction... *Shrug*
And how much storage does that $1000 buy you anyway? Can you toss in 2TB drives to upgrade the storage? Does the controller support that? Are they SATA or SCSI? In other words does $1000 buy you a usable and *upgradeable* large data solution, or is it a one shot deal?-JavaJones
You can get either SATA or SCSI (SCSI typically being cheaper and smaller these days). But get the one you already have drives for, all you need are the drive trays ($10 - $20 each) and you can use the existing drives you have now. You're only limit is the number of Hot-Swap drive slots (6, 8, 10, more) there are on the machine. Yes the 2TB (SATA, I'm not sure on SCSI - probably if you can afford them, I can't) drives will work.
As for my own situation with the Lacie unit, it's getting a bit off-topic but your proposed solution isn't really applicable to me either. I dunno about you, but I don't equate a stand-alone unit I can just plug in to a USB/eSATA/Firewire port with a complete, separate machine with RAID. I guess I could just use that machine as a sort of NAS, but the whole point to me was to have *local data access speeds* over e.g. eSATA, rather than network-limited speeds.-JavaJones
Hm... GBit limited Network speeds. How often are you moving what size file(s)? It's not like browsing the filesystem is going to lag with the traffic of a home LAN (which everybody has these days - Even if they don't know it). How many of those Units do you have in order to be able to store the 20TB of stuff you mentioned having? They can't be that cheap.
-
+1 on that. I've seldom seen a rebuild go 100% smoothly. Usually all of the drives were purchased at the same time
-40hz
But the solution, surely, is just don't do that. RAID reliability assumes the probability of one drive failing is independant of the others. While that ideal can't be met in practice, you can get close to it by buying different drives from different sources and ideally different manufacturers.
-
@Eóin - you are quite correct. But I wasn't referring so much to the manufacturing batch numbers as I was to them likely all being on the same engineering revision, equally old, and equally (ab)used in whatever environment they're in.
In my experience, its environmental factors and age that have the most bearing on a drive's useful life expectancy. I haven't run into too many manufacturer's defects when it comes to server drives.
-
Hm... GBit limited Network speeds. How often are you moving what size file(s)? It's not like browsing the filesystem is going to lag with the traffic of a home LAN (which everybody has these days - Even if they don't know it).-Stoic Joker
Todays' drives are going to give you ~100MB/s transfer speeds - a typical gbit LAN speed is closer to 30-40MB/s, and with a lot worse latency as well. Quite a difference if you're shuffling large media files around :)
-
Hm... GBit limited Network speeds. How often are you moving what size file(s)? It's not like browsing the filesystem is going to lag with the traffic of a home LAN (which everybody has these days - Even if they don't know it).-Stoic Joker
Todays' drives are going to give you ~100MB/s transfer speeds - a typical gbit LAN speed is closer to 30-40MB/s, and with a lot worse latency as well. Quite a difference if you're shuffling large media files around :)
-f0dder
right, knew that part ... That's why I was pondering his usage/habits. I also used the word browsing for that reason. If moving a music CD the speed difference isn't a big deal. Now if you're moving a 25GB BlueRay .iso, well then yeah...Speed is a factor.
So if everything is scattered about in little data huts that would be a huge (mess) impact. But if you RDP into a server to move media files around locally there ... (hehe) ... Zoom! Which is what I do with the ton of (OS and software) ISOs I have. Granted I don't have to futz with them frequently, but when I do it never takes long.
Anyhow the question is how often does a huge media need to be moved local <-> remote? Both viewing and burning can be done over the wire. So (I am guessing here) really it's just a case of incoming organization and the occasional cleanup of stuff nobody needs/likes anymore.
-
I've had a NAS for many years. Set up with hardware based RAID. I've also used backup programs for many years.
Increasingly, though, I have become concerned about how much security they really bring. I've not had a failure with either (yet); at least, that I know about. And it is knowing about it that I really want to do. A backup is always a worry unless you check every time that it has everything and you can read it. I'm now much happier just syncing all my data files, and periodically copying them in their entirety to other drives/places (with copies of various stuff in places on the net). It is very easy to copy TrueCrypt volumes too. It means I can check or use either set of files in normal use; I always sync in one direction only (feeling that gives me a bit of extra protection against something going wrong), so I have to remember to save everything on to one drive only, but that is not a problem. I also image my OS/App drive regularly. I know most of them are OK because I do check from time to time, and I can always rebuild from scratch if necessary anyway.
And my NAS? Well, when I get around to it I will just copy everything off it, reformat it without RAID and use it as separate drives. I just don't see that RAID is bringing me anything I need right now. With the current price and speed of usb3/sata3 drives & SSDs, it seems best to just add them as I need them.
PS I voted for FreeNAS, because that is all I would use a home server for. If I wanted a real server, I'd go Linux. Much more familiar with Linux servers than Windows Servers.
-
In my experience, its environmental factors and age that have the most bearing on a drive's useful life expectancy.-40hz
Agreed, but that doesn't necessarily mean they'll be likely to fail like dominos. Our DC here is about 7 years old, two of the 3 drives in it failed 8 months apart about 2 years ago. Assuming the third should be failing shortly I ordered two drives when the second one failed ... I'm still sitting on the spair as disk three ain't even blinked yet. *Shrug*
We also have an HP Netserver LH3 (Dual PII 400s - WWwweeeeeeeeee) that was purchased back when the company first opened almost 20 years ago (It was originally a Novell box). It still has two of it's original 3 drives in it running Server 2k3 24/7 (and it once went almost 2 years without a reboot). I ordered it's replacement today (Dell PowerEdge 2800) soley because it's just too old for the latest update of the accounting software (which is all it does). I'll probably keep it running in the lab just to see which one of us dies first... :)
-
^What can I say? Your individual performance and mileage may vary. :)
Got a Northgate 386-20 with a 80Mb drive in it that still boots up WFWG on cue everytime I hit the switch. I've had two server drives go south on me twice (different machines) in the last 6 months. Top quality units too. Both were members of RAID arrays. Both cases encountered a second member drive failure - one during the rebuild (RAID-1); the second (RAID-5) one month and 4 days after the rebuild.
Saw that exact same thing happen once the year before that.
Please underdtand I'm not trying to read too much into it. We all know drives fail all the time.
But it has made me reassess my opinion of exactly what RAID brings to the table. And also what representations I'm making to my clients about it.
-
I hear ya... I had a client with 4 drives in a RAID5 array, one drive failed ... but nobody said anything. I got there 3 months later after they'd been screwing with it for awhile. I managed to get it to rebuild and the box came back online. There were errors showing in the logs that another disk was failing, so I shut the box back down and told them not to run it until we got some replacements. They did not listen and flipped the thing back on the instant I left the parking lot. Three days (of constant use) later two more of the drives failed.
To the greatest of plans there is always the perfect flaw...
Whoever set up their backups did it wrong. Data Was UnRecoverable.
But other than that my experience with RAID has been positive.
-
Stoic, one of the things you seem to be implying above is that RAID can be a stand-in for backup (the comment about how do you backup 10TB of data). As far as I understand that's contrary to common wisdom about RAID. Working on that assumption then, even if you do have it in a RAID, you should back it up anyway, in which case if the main benefit of RAID is its redundancy, then a good sync/backup system gives you the same benefit. Hence why I don't really see the value of RAID in a low-cost, home user environment. For enterprises where they can afford to have both high-end RAID *and* a good backup system, and where high uptime is a requirement, RAID absolutely makes sense.
When I said "complexity" I wasn't necessarily referring to the end-user interaction and maintenance, more to the complexity of the *system* (and I think you significantly overestimate the complexity of a properly setup syncing system - it's essentially automatic and largely maintenance free). A RAID setup is inherently more complex in terms of hardware than a bunch of independent disks running off SATA on the motherboard (for example). The more "moving parts" and interconnectedness, the more a failure in any part of the system can affect the whole, even if redundancy is a part of the architecture (as it is in RAID). A prime example of that is the battery backup for cache operations on a RAID controller. What if the battery fails, does the user know how to replace it? (and with older off-lease equipment wouldn't it be even more likely to fail?) And if the power goes out while the battery is dead, the whole array could be corrupted. Power goes out with a stand-alone SATA drive and the worst thing that could happen is the drive that is being accessed at that moment has a bit of corruption. In 15 years of working with home and small business PCs I've not seen anything more than minor file corruption from power loss issues.
As to my data setup and speeds of access, the 10-20TB are on my media "server" and I don't usually access those remotely. I watch movies through my HDTV straight from the media machine. The only thing I will occasionally stream to other devices in the house is music, or maybe a TV episode (usually just do Netflix for that though). If I had more TVs in the house this might be more of a concern.
Anyway, what I do with the Lacie 4big is quite different. I have all my digital photos (about 60,000, around 300GB), my work files, project files, Terragen files (100GB or so), etc. on there. In truth I only have about 1.1TB of that filled up. And that's actually the most important data for me to back up. But access speed is important because I work with large image files frequently, and don't want to be pushing and pulling them over the network all the time. Imagine browsing an image catalog of 20+MB images. Flick through a few images and you'll see slowdown on the network, but with a modern, fast drive, on local access the speed is fine. I can't speak for superboyac's needs as far as backup/redundancy or external access on his proposed 20TB of data, but if he's accessing it over the network, any speed benefit of RAID will be lost anyway.
- Oshyan
-
Stoic, one of the things you seem to be implying above is that RAID can be a stand-in for backup (the comment about how do you backup 10TB of data). As far as I understand that's contrary to common wisdom about RAID.-JavaJones
Many of the really high-end deduplicating RAID6 storage devices claim to be able to eliminate the need for a backup. However I'm not entirely sold on that idea either. I prefer to have RAID as a first line of defense, and then have a backup for those "special moments" when shit really hits the fan.
Here's the thing, from SMBs to SOHOs down to the average Home Owner, everybody is on-a-budget. So what typically happens is enough hardware is bought to store the data being used. Some type of backup is discussed and is maybe even used a few times. Then the data grows and the space needed to store it gets bigger. Now it all no-longer fits on the backup, so the "important stuff" discussion ensues. Either that or somebody comes up with the "Let's do an incremental backup" idea, and nobody has the forethought to shoot them for saying it ... before the plan is put in place.
Note: I'm cringing right now just thinking about the number of time I walked into a new client to find they've been using the same five pieces of media to do incremental backups, for the last five years ... Chance of a successful recovery? Zero!
Both scenarios tend to end badly as backups just love to fail. And they do so at an extremely high rate. RAID on the other hand, could fail. Synchronization & backup schemes both have the same (glaring) flaw. The are simple, easy, set-it-and-forget-it systems. ...and it's that last, forget it, part that bites'em in the ass every time!
Synchronization, like mirroring (RAID1) both like to give people two identical copies of the same error. So if a disk fails in the middle of a sync... (point here being every system has a weak point)
Now for the average end user that comes trotting home with a brand new PC, a few ideas of what they can do with it, and maybe even a shiny new digital camera... Obviously a server would be a completely stupid to even suggest. (I'm picturing Goofy from the old Disney driving cartoons trotting out of a BestBuy here)
However, you and SB are not typical end users. You have an established system architected into a strategy that is so far working for you. Great go with it. SB on the other hand, is looking to start-from-scratch (so to speak), and is coming out of the gate planing on warehousing 10+TB of data. Those NAS boxes ain't cheap, as anything with more than 2-3 slots is well over $1,000. So if you looking at budgeting enough money for three of those ... You are much better off going with the off-lease commercial hardware that will give you a true scale-able hardware RAID configuration for a fraction of the money you would have spent on the consumer grade stuff.
Even if you are working out of your mother's basement (not saying you guys are), with 10+TB of data you have an enterprise class problem ... Which should be handled in much the same way regardless of where the data is and whom it belongs.
I'm hoping I covered everything as this has takes awhile for me to type. :)
-
Thanks Stoic, believe me, i read every word of that.
What I need to do is sit down and visually map out my backup strategy. Once I do that, I can post it here and everyone can debate all the weak points. But before I do that, and I imagine it will take me a while at the rate I'm going, I do have a few main points to discuss. Please keep in mind that I do not understand everything I am about to say in the detail I am used to understanding things before I act on it.
--I STILL do not understand how RAID is useful to me. I'm not trying to backup an OS or anything. Just standalone files and folders (mostly media: music, video, docs, etc.). I still consider RAID more of a performance thing than a backup. I get the rebuilding thing, but I don't see any advantage compared to just copying all my files/folders onto a fresh hard drive. That, to me, is the same thing as "rebuilding" a drive. The benefit is that the physical drive is a standalone unit, which I really desire. I LOVE the idea of pulling a drive out at anytime and using it in any other computer. I don't like the idea of RAID for my purposes.
--The one thing I am worried about and will probably spend the most time thinking about: how to prevent BAD backups and only have good backups. Let me explain. Let's say I have a hard drive, and I back it up with double-redundancy using two more hard drives and syncing (SFFS). Now, let's say the original drive got infected with a virus that got into a bunch of my files. Soon after, that virus will be backed up to three places. And now I don't have an original GOOD version anywhere! How do you prevent that? It doesn't help if you back up with multiple redundancy if they all have the same BAD files.
So then I think, well, the backups should be staggered somewhat. In other words, maybe one drive backs something up once a week, and the other once a month. That's good, right? Eh...it's ok. I'd like something better.
Then we start thinking incremental backups or versioning. Well, I'm not really a big fan of that either because, for one thing, the files are not standalone anymore. You're going to need some kind of software or something to extract whatever file you want from the backup. And again, it's not a exact copy of the files/folders...it's more like an archive of files/folders. The other problem I have with it is I can't incremental/version everything. Otherwise it would be too big. So then I have to think about what's important and what's not, and I don't want to do that.
So I just don't know what the optimal way to backup things is. I'm sure all of this has been well thought out for big companies. I can't imagine big companies not having something rock solid and that covers all bases. But then again, I'm never surprised anymore if that's not the case.
I just don't know what the perfect solution is. I've read mouser's long backup document, and I even wrote a shorter one here myself. But there are too many holes right now in my strategy, and furthermore, it's a puzzle that I haven't thought through long enough.
Cmon people, we're talking here as if it's the first time we've all ever thought about backing up. With all the computer people here, and professional IT people, someone MUST have a really solid solution that is already in practice, no? Am I wrong?
-
Just to clarify, I wasn't suggesting SB get a bunch of NAS units. The whole conversation has centered around a single "server" machine, I was just assuming more drives in the machine to handle redundancy, as SB himself had suggested. I think when I make my 2nd blog post about my backup strategy some of my thinking on all this will become more clear. My opinion has changed a bit since my big drive failure. :D
Unfortunately there's no easy way around how to *fully* back up 10+TB of data, I agree. Fortunately I don't have that much data I really need to back up. It sounds like SB maybe does, so that's a conundrum. I could easily suggest a pretty reasonable backup strategy that would be "secure" and "redundant". It would just take a lot of hardware, heh. I wouldn't normally see RAID as necessary, but it might play a part solely for the simplicity it brings in addressing large contiguous disk space. Otherwise it just adds cost without really adding a worthwhile level of redundancy (IMHO, I'll just leave it at that).
As far as the idea of having to determine what's important and what's not, I think this kind of has to be a part of any backup strategy. For example some people (like myself) don't really care about backing up the whole OS, but want all the data, and that in itself is a decision about data importance that's made. Going the next step and realizing that you don't necessarily *need* triple redundancy on your archive of ripped DVD movies since you have the originals is a reasonable and important thing to do. It saves you having to back up lots of large data. I've made this call about my media archive, because I can always re-rip or re-download something. I can never re-take a photo and re-writing a document will never be the same, so that stuff is worth triple redundancy if possible.
If you really don't want to think about it and have the money and hardware space, just get 3x the number of drives you need, use basic sync/mirror for one data set, and a versioning backup program (like CrashPlan) for the other. That way you have a normal copy of the data that doesn't depend on a backup app functioning correctly (the sync/mirror) and a versioned backup that should help in the case of e.g. viruses, etc. Of course this isn't a truly bulletproof solution, first because there's no off-site backup, and second because there's only one versioned setup and if that fails and you need versioning, you may be hosed. But I consider that a pretty outside case.
Off-site backup of 10+TB of data is sadly not terribly practical. But again there are solutions to it if it's truly important. That's a situation where I really do think an external NAS-type box (but actually a locally connected one, e.g. eSATA, not network-attached) would make sense. You can get external RAID boxes that support 5 or more drives, drop 2TB units in there, and just RAID0 it so you have a big contiguous drive to back up to. RAID0 is inherently more dangerous of course, but since this is just 1 of 2 or 3 backup locations, it's ok. It's really just there for catastrophic events (e.g. your house burns down, taking your other 2 backups with it). If you have the additional drive space to do so, might as well RAID5 of course.
Note I am only suggesting RAID here because it makes it easy to address it as one big unit, not because of redundancy. I'm not sure how difficult it really is to externally address *multiple* SATA drives through eSATA, though I know it's possible (and there have been some discussions here about it). If it's possible with some of the existing external SATA drive enclosures, then it might be preferable just to have them as an additional sync target, not in a RAID but as individual drives.
Anyway, if we assume 10TB and that securely and redundantly backing up *all* of it is desired, I'd therefore suggest the following, summarized from above:
- 1 system with 10 2TB SATA drives
- An automated sync or mirroring between 1 set of 5 internal 2TB drives to the other set
- 2 external drive enclosures, one with 5x2TB drives the other with more if possible (this would be the incremental backup drive)
- Use one external enclosure in simple disk mode if possible, doing an additional sync of all drives to the external unit once a week, then take it off-site (e.g. to a storage unit, safe deposit box, etc.)
- Use the other external enclosure in RAID (RAID5 if possible) and use an incremental backup process on the internal drives targeting the external unit
This setup is pretty overkill and expensive, but will give you triple data redundancy *with* off-site backup *and* versioning. If you don't care about the off-site backup, then just remove that part from the equation and simplify. :)
- Oshyan
-
It's a question of static vs. dynamic data. Backup frequency is dependant on how often critical changes are made.
Example:
I have a client with 300GB of important business data, and a 36GB tape backup drive. But, 90% of the data is historical (e.g. never changes). So... The have 2 servers, one new, and the old one it replaced.
Server 1 (the new(er) one), runs SBS2k3, and stores all their currently active stuff. It also has the tape drive in it. Backup of their active data, System State, AD, etc. require all of one tape, and sometimes part of a second. Exchange (also critical to backup) just flat won't fit...without going to a 3 tape a day rotation - Which is insane. So...
Server 1 is RAID5, That's the primary line of defense for the OS in the event of a disk failure. secondary line (if 2 disk failure, etc.), is the daily backup of AD and System State data. The complete configuration is also heavily documented so a brick level rebuild can be done quickly if necessary. The other business data is backed up fully Monday through Thursday when a 2nd tape can be added in the morning if necessary (this is the data that changes most frequently).
Exchange is backed up on Fridays only by itself, because it handily fits on one tape, and nobody is there on Saturdays to add a second tape. otherwise a daily business data backup would always fail (timeout) on Saturday morning if it was run. So there I've staged when what was backed based on a heriarchy of what was most likely to change to maximize the highest number of changes that could be saved.
Now the bulk of the static historical data on server 2 (also RAID5) is backed up once a month, to one of a pair of 1.5TB USB drives. This is because the data on it only changes when the file handling policy declares that something is "old enough" and is manually pushed to the server. This is done in the last few days of the month, because the USB drive backup is scheduled on the 1st of every month.
That way if - by some odd strike of bad luck - something went wrong big time ... the only unbackedup data that could be lost, would be the most current, and therefore easiest data to recreate.
All of the backup operations are logged, and a full report of the results are Emailed to both myself and the business' owner every morning. That way if something does go wrong with the backups... We Know! ;)
-
Damn JJ we're turning into a couple of long winded bastards here ain't we? I'm guessing you type alot faster than I do 'cause ya beat me ... So I'll have to read yours in a minute.
:D
-
Hehe, I have a pretty decent typing speed. I'm definitely long-winded, which is probably part of why I've become a fast typer. ;)
Personally, though I used to use tape backup more than a decade ago, I just don't see myself ever going back there given the low cost of drive space and speed advantages (both for backup and restore). I know it's still widely used by enterprise though.
- Oshyan
-
Personally, though I used to use tape backup more than a decade ago, I just don't see myself ever going back there given the low cost of drive space and speed advantages (both for backup and restore). I know it's still widely used by enterprise though.-JavaJones
I'm with Ya, I'm beginning to transition to USB drive backups myself now that the small easy to carry 2.5" drives are over 500GB and under $100. However the setup I described above was deployed 5 years ago when that wasn't the case. So tape made sense back then.
My point for SB was how the backups can be staggard to maximize coverage and minimize futzing arround at the same time. By organizing the data based on (static vs. dynamic) change frequency. And I agree that not everything really needs to be backed up if it's already stored as a hard copy, or just really not that critical (we all have a few files like that...).
-
--The one thing I am worried about and will probably spend the most time thinking about: how to prevent BAD backups and only have good backups. Let me explain. Let's say I have a hard drive, and I back it up with double-redundancy using two more hard drives and syncing (SFFS). Now, let's say the original drive got infected with a virus that got into a bunch of my files. Soon after, that virus will be backed up to three places. And now I don't have an original GOOD version anywhere! How do you prevent that? It doesn't help if you back up with multiple redundancy if they all have the same BAD files.-superboyac
If you become infected, you should expect all files on all disks on your system to be infected... but (silent) file corruption not caused by virii is another situation.
You could do a staggered setup, but that's going to require a fair amount of disk space, and will probably be a maintenance nightmare.
I don't know a Windows equivalent, but rsnapshot (http://rsnapshot.org/) is very nifty. It does incremental backups, but not in the traditional sense... take a look :)
-
--The one thing I am worried about and will probably spend the most time thinking about: how to prevent BAD backups and only have good backups. Let me explain. Let's say I have a hard drive, and I back it up with double-redundancy using two more hard drives and syncing (SFFS). Now, let's say the original drive got infected with a virus that got into a bunch of my files. Soon after, that virus will be backed up to three places. And now I don't have an original GOOD version anywhere! How do you prevent that? It doesn't help if you back up with multiple redundancy if they all have the same BAD files.-superboyac
If you become infected, you should expect all files on all disks on your system to be infected... but (silent) file corruption not caused by virii is another situation.-f0dder
Realistically how likely of a threat is that (I've been wondering about this for a while)? It seems like the bulk of the EvilDoerWare these days only goes after the shell and your wallet/identity (not necessarily in that order). There really hasn't been a good file eating virus (that I recall) in the wild since Snow White almost a decade ago. The SBD file corruption seem a more (real?) likely threat these days to me (But I'm on the fence).
Mind you I'm not arguing the point, I really am just curious about your thoughts on this matter.
-
I don't know a Windows equivalent, but rsnapshot (http://rsnapshot.org/) is very nifty. It does incremental backups, but not in the traditional sense... take a look :)-f0dder
Dammit! I'm positive I found one on the net recently that was based upon this blog article: An intelligent backup system for Windows (http://mutable.net/blog/archive/2006/11/13/an-intelligent-backup-system-for-windows-part-1.aspx) (4 part article)
Do you think I can find it again....nope!
Found it: DataSafe Backup (http://www.sofgem.com/products/datasafe.html)
Lucky I never throw away anything I download :P
-
If you become infected, you should expect all files on all disks on your system to be infected... but (silent) file corruption not caused by virii is another situation.-f0dder
Realistically how likely of a threat is that (I've been wondering about this for a while)? It seems like the bulk of the EvilDoerWare these days only goes after the shell and your wallet/identity (not necessarily in that order). There really hasn't been a good file eating virus (that I recall) in the wild since Snow White almost a decade ago. The SBD file corruption seem a more (real?) likely threat these days to me (But I'm on the fence).-Stoic Joker
Dunno if you're likely to run into a .exe infector these days, like you I've got the feeling (though not scientifically backed!) that they're a thing of the past.
There's been several examples of ransomware the last few years, though, and those go after all your files, not just executables...
-
I don't know a Windows equivalent, but rsnapshot is very nifty. It does incremental backups, but not in the traditional sense... take a look
Had to dig into our KB for this one: There is something called cwRsync (http://www.itefix.no/i2/node/10650) that will allow you to easily integrate the rsync daemon with various flavors of Windows. Catch is the Windows rsync port requires Cygwin. Can't vouch for how well it actually works since I've never tried it. Maybe it's because I'm not that wild about loading Cygwin on a server. (S'truth I'm not too wild about Cygwin in general, but that's probably just me.)
cwRsync is a yet another packaging of Rsync and Cygwin. You can use cwRsync for fast remote file backup and synchronization. Rsync uses the Rsync algorithm which provides a very fast method for bringing remote files into sync. It does this by sending just the differences in the files across the link, without requiring that both sets of files are present at one of the ends of the link beforehand. At first glance this may seem impossible because the calculation of diffs between two files normally requires local access to both files.
Rsync normally uses ssh for communication. It requires no special privileges for installation. You must, however, have a working ssh system.
Alternatively, rsync can run in `daemon' mode, listening on a socket. This is generally used for public file distribution, although authentication and access control are available. Cygwin is a Linux-like environment for Windows. It consists of a DLL (cygwin1.dll), which emulates substantial Linux API functionality, and a collection of tools.
Installation
Supported platforms: Client - NT/2000/XP/2003/2008/Vista/7, Server - NT/2000/XP/2003/2008/Vista/7.
If anybody tries it please let me know?
-
@4wd - do you know if DataSafe also backs up servers: or just workstations from the server?
-
@4wd - do you know if DataSafe also backs up servers: or just workstations from the server?-40hz
Do you mean does it work like rsync, where you need a client/server relationship?
Then no, it's more like your traditional Windows backup program, (ala SyncBack, etc), choose a source and destination, (which can be FTP, CD or Network Share), choose your options, (Full, Incremental, Differential, Mirror; Compression, Encryption, Versions, etc) and hit go.
If you choose a backup type of Mirror, then Hardlinks are enabled.
I haven't extensively tested it, (I've got a few small OS gremlins that I need a re-install to fix, eg. VSS isn't working), but it seemed to work OK.
eg.
1st Run - 472 jpgs/pngs (1.78GB) copied in 32s (HDD -> HDD)
2nd Run - no changes, 472 files copied in 4.5s via hardlinks (2 or 3 seconds is because it tries to get VSS working)
3rd Run - Added another 51 jpgs, 523 files copied in 6.9s, 472 via hardlinks
Each backup run goes into it's own sub-directory named Jobname.backuptype.runnumber, (eg. Test.Mirror.00008), and then the full directory structure of what was backed up.
I haven't tested it to my FreeNAS shares, (UFS), might try that tomorrow, (technically already tomorrow here but it's 0433 :) ).
Update: Works fine to FreeNAS shares on UFS drives, happily creates hardlinks for unchanged files which reduces space requirements a fair bit.
It's quite amazing, (well to me who hasn't had experience with links), seeing six (6) backups of the exact same files and not see the available disk space drop. Effectively, 2GB of space used to store 10GB of uncompressed data - very neat.
I think I've just created a conflict in my brain....I like the speed/versioning of DataSafe but I have SyncBack Pro :(
Must...resolve...conflict........aaarrgghhhh! <- Brain implosion.
There is something called cwRsync (http://www.itefix.no/i2/node/10650) that will allow you to easily integrate the rsync daemon with various flavors of Windows. Catch is the Windows rsync port requires Cygwin.-40hz
There's also DeltaCopy (http://www.aboutmyip.com/AboutMyXApp/DeltaCopy.jsp).
In general terms, DeltaCopy is an open source, fast incremental backup program. Let's say you have to backup one file that is 500 MB every night. A normal file copy would copy the entire file even if a few bytes have changed. DeltaCopy, on the other hand, would only copy the part of file that has actually been modified. This reduces the data transfer to just a small fraction of 500 MB saving time and network bandwidth.
In technical terms, DeltaCopy is a "Windows Friendly" wrapper around the Rsync program, currently maintained by Wayne Davison. "rsync" is primarily designed for Unix/Linux/BSD systems. Although ports are available for Windows, they typically require downloading Cygwin libraries and manual configuration.
DeltaCopy doesn't require CygWin, all Client/Server binaries are in the one installation binary and it's very easy to set up.