Help me overbuild my home network

[Vurbal]

I'm gearing up for a major home network upgrade, actually more like building a completely new network from the ground up, and I need to tap into some of DC's IT wisdom to make the leap from concept to design. Originally my plan was simply to build a networked DVR and media server using MythTV (LinHES technically) to exorcise my house of Hulu Plus. Naturally I couldn't just build a server and put in some set-top boxes. Being... well me... I couldn't stick with something so straight forward and simple and now I've managed to escalate my plans into something truly batshit crazy.

Last week, when I was at a used office furniture store buying a desk, I lucked into a deal on a server cabinet so now I'm pretty well committed. Based on my wife's reaction when I put the 7 foot behemoth in my office (did I mention it's a full 42u cabinet?), she's all in favor of having me committed too. Hopefully if I pull this off I'll get to keep sleeping in the same room with her for many years to come.

So here's the new plan. Instead of nice new hardware I've decided to take advantage of the huge volume of outdated HP enterprise (well low end enterprise) servers available for next to nothing. Specifically I'm focusing on Generation 5 Proliant dl360 units which I should cost inside of $250 each including rack rails and shipping. I'm planning 3 servers - 1 for a border/network appliance, 1 for management (OpenLDAP mostly) and VM testing and one for media and possibly a couple other services once I'm comfortable with the performance.

The servers are going to be running ProxMox VE for virtualization and some combination of KVM and OpenVZ for the VMs. Since they will all be sharing drive space, which I'll also be using to expand my desktop, most of my spending is going to be on storage and network infrastructure. I'm going to want something more than a simple NAS so I've decided to setup a SAN instead. To ensure smooth network traffic and good throughput I'll be putting either 2 or 4 port Intel server NICs in all the machines, including the SAN.

Now I have to figure out the SAN hardware. Ideally I'd be using 2.5 inch SAS drives but they're just too expensive. Instead I'll be going with 3.5 inch SAS, Seagate Constellation ES specifically. They're designed for storage arrays and have a 5 year warranty. I'll be using either 1TB or 2TB drives in a RAID 6 configuration and a hardware controller with at least 512MB of BBWC. It will be either 6 or 7 drives which is more than Seagate recommends (they say up to 5) but I don't see that being a problem.

What's missing now is figuring out a controller to use and a server platform to put it in. Older hardware is a problem in this case because I don't want to settle for SAS-1 speed when I have the faster drives. There's already going to be a performance hit from the redundant parity for RAID 6. Ideally I want to set it up to provide some private cloud services for things like photo viewing and a file locker. Basically I want it to mesh well with my wife's iPhone similar to how public cloud services work. That should take care of her annoyance at the whole project.

That just leaves the issue of traffic management. I'd like to find a good used managed switch. Unfortunately managed managed switches are one of the last real scams left in the low end enterprise market. I mostly blame Cisco for that. Fortunately it seems like I can pick up an older ProCurve Chassis and the requisite cards for around $200. I also need to do some work on the wireless coverage around the house, and especially out in the garage, but I'll hold off on that until everything else is good and stable.

Any thoughts? Suggestions? Questions?

Feel free to just point and laugh. I would.

[x16wda]

On the plus side, a 42U rack will have a seating capacity of at least 1 even with the hardware installed, if you start at the top! 

It still looks like you will be spending a fair amount, can't you just spring for a couple extra drives and do RAID 10?  That cuts the parity crap and extra writes out of the equation and ought to result in better performance.

[Innuendo]

Sometimes you can get lucky finding stuff on eBay. A couple years ago my network switch died and I needed a new one. Found a nearly new D-Link DGS-1224T Rev C 24-port managed switch on eBay for the outrageous price of $70 delivered.

[Vurbal]

Sometimes you can get lucky finding stuff on eBay. A couple years ago my network switch died and I needed a new one. Found a nearly new D-Link DGS-1224T Rev C 24-port managed switch on eBay for the outrageous price of $70 delivered.

Yeah, I'm planning to spend at least a week or 2 looking daily in case something pops up. The more I think about it and the more I look on eBay, though, the less concerned I am. Modular units like some of the old 8 card HP models should be easy enough to come by. They're still perfectly good but big enterprise operations don't have any use for them today because they fall short on both ports and internal bandwidth.

OTOH I'm not in any danger of even approaching either of those problems. Since it's going to be hidden away in a cabinet anyway it won't even be an eyesore having a mostly empty chassis around. I'd still jump at a deal on something newer and better suited to my needs but in the long run it shouldn't be a big deal either way.

[Vurbal]

On the plus side, a 42U rack will have a seating capacity of at least 1 even with the hardware installed, if you start at the top! 

I'll have to put the chair at the bottom. I wouldn't want my long hair getting sucked into the exhaust fan at the top.

It still looks like you will be spending a fair amount, can't you just spring for a couple extra drives and do RAID 10?  That cuts the parity crap and extra writes out of the equation and ought to result in better performance.

I'm definitely going to be doing some significant spending, a little more than I would have under my original plan. I'll just be getting a lot more for the money, and honestly most of it will probably outlast whatever I would have put together before.

At the end of the day, though, 2 more drives wouldn't significantly change the factors that led me to choose RAID 6. A lot of it boils down to the fact I've gone a long time without increasing my storage significantly. I already have projects in the works that will push my needs well past what I have now. I foresee needing to add at least another 2TB within the next year cheaply enough that I can also get a lot more backup capacity in place.

The price of things like a good RAID controller or high end NICs doesn't really bother me since I look at them as just an upfront cost that will pay dividends for a long time. Even if one of these servers dies, they will likely reduce the replacement cost. At this point the worst case scenario would be deciding SAS-2 hardware isn't worth the price right now and falling back on SAS-1. It seems likely I won't be pushing the boundaries of SAS-1 with anything I've got planned.

Assuming I go that route, I could go with a lower end Proliant than the dl360 and one of HP's P800 RAID controllers. Add in a MSA60 drive shelf and the price would still probably be less than the controller itself would run me for SAS-2. That's after the expense of buying new cache batteries for the P800. That may not be necessary but I'd do it on general principle anyway.

[Stoic Joker]

To ensure smooth network traffic and good throughput I'll be putting either 2 or 4 port Intel server NICs in all the machines, including the SAN.

That's not going to give you what you're after. Teaming 4 NICs won't give you a 4Gb pipe for single transfers. It will give 4 1Gb pipes to 4 different transfers. If you want maximum speed across the backbone, go either fiber or Cat6a 10Gb copper.

Now I have to figure out the SAN hardware. Ideally I'd be using 2.5 inch SAS drives but they're just too expensive. Instead I'll be going with 3.5 inch SAS, Seagate Constellation ES specifically. They're designed for storage arrays and have a 5 year warranty. I'll be using either 1TB or 2TB drives in a RAID 6 configuration and a hardware controller with at least 512MB of BBWC. It will be either 6 or 7 drives which is more than Seagate recommends (they say up to 5) but I don't see that being a problem.

RAID6? ...That's even overkill by my standards. The 2.5 drives are cheaper to run, so go with 2.5" 7,200 Nearline SAS drives to save cost and still get the 6GB transfer speeds.

I hope that's an enclosed cabinet...because rack systems do tend to be very loud ... And hot. Make plans for keeping the thing cool.

[Vurbal]

First of all I should add some context to the setup detailed in the first post. It's not final by any means. It's a middle step in my (yes, very messy and inefficient) design process. Some of the components (like the server NICs and switch) are from different steps in the process so they may not make sense together.

I've just reached a point where I need more information about some of the technology from people with personal knowledge. When I left IT I was dealing with G3 Proliant DL servers and just a 1GB fiber backbone. Virtualization hadn't really gotten down to our level. Obviously SANs weren't a thing for us either. I have a lot of catching up to do still. I haven't even started on reading the documentation for the SAN (probably FreeNAS or NAS4Free) or virtualization software.

Normally at this point I would probably still have at least a week of full time research left before going back to the beginning and building the design from scratch. Like I said, it's a chaotic process. Thankfully DC is the rare place I feel confident I'll find someone to point me in the right direction and save days. The more critical anyone wants to be, the more useful it is for me.

To ensure smooth network traffic and good throughput I'll be putting either 2 or 4 port Intel server NICs in all the machines, including the SAN.

That's not going to give you what you're after. Teaming 4 NICs won't give you a 4Gb pipe for single transfers. It will give 4 1Gb pipes to 4 different transfers. If you want maximum speed across the backbone, go either fiber or Cat6a 10Gb copper.

Now I have to figure out the SAN hardware. Ideally I'd be using 2.5 inch SAS drives but they're just too expensive. Instead I'll be going with 3.5 inch SAS, Seagate Constellation ES specifically. They're designed for storage arrays and have a 5 year warranty. I'll be using either 1TB or 2TB drives in a RAID 6 configuration and a hardware controller with at least 512MB of BBWC. It will be either 6 or 7 drives which is more than Seagate recommends (they say up to 5) but I don't see that being a problem.

RAID6? ...That's even overkill by my standards. The 2.5 drives are cheaper to run, so go with 2.5" 7,200 Nearline SAS drives to save cost and still get the 6GB transfer speeds.

The Constellation drives are nearline SAS - well actually I think that's all the 2.5 inch models and everything 2TB and up for 3.5 inches. Admittedly I did minimal research and signigicantly less comparison shopping.

I think it's time to back up a little bit and make myself a logical map of the servers/services and clients I'm going to have. It occurs to me that, considering my limited experience with VMs, I may be completely off about the hardware requirements. Eliminating a server could drastically improve my storage budget, making 2.5 inch drives perhaps a viable option. And the more I think about it, the more I think I'd be crazy not to go with RAID 10. If I happen to end up with a server equipped with a good RAID card, is there any argument for using it instead of software RAID 10?

I think I know enough now that I should make a logical diagram of the servers/services and clients that will be on the network. Then I can try to sort out how to best host them as virtual machines and, potentially, what I don't want to virtualize. I'm still kind of stuck in the world where you were concerned that certain services run from different physical servers.

I suspect for my network the only serious consideration should be avoiding physical resource bottlenecks. Until I can estimate resource demands only wild overestimates are safe. I intend to overshoot my needs, but by a foot or 2, not a mile. I have a feeling when I see all the pieces together I'll at least be able to formulate the problem better.

I hope that's an enclosed cabinet...because rack systems do tend to be very loud ... And hot. Make plans for keeping the thing cool.

It's enclosed and actually not terrible looking for an office. It even has a smoked glass window in the door. I wouldn't have considered an open rack, even with just a few servers. The company that hired me to be their first network admin had the brilliant idea that I could work from a table in the server room. It only had one cabinet, nowhere near half full, but the back door was off, the room was (literally) an old supply room with no serious cooling. It was loud, but damn was it hot!

Soundproofing shouldn't be too hard. For the noise level I'll be dealing with, acoustic foam inside the cabinet should be good enough. Otherwise I have it set up with walls or surfaces on 3 sides which could all get the same treatment.

For the moment, cooling is a series of escalating measures I may have to use. My office has nearly 9 foot ceilings and there's a ceiling fan positioned at a good height and distance to circulate exhaust heat from the top of the cabinet. There's a vent a couple feet in front of it, and all the ac/heat vents in the house have individual dampers. In a worst case scenario the window in front of it would get a window AC unit. Of course that introduces the noise problem again. 

On one hand it would have been nice to get a 21u cabinet instead, but my only actual option was a 12u cabinet on wheels and it looked like cooling would be a problem. It certainly wouldn't be helped significantly by the ceiling fan.