Please help superboyac build a server (2013 edition).

[superboyac]

Modified 8-15-2013
Mini-Me Test-Run Storage Server:

ITEM	BRAND	MODEL	PRICE	STORE
Motherboard	SuperMicro	X10SAE (ATX form factor)	$216	===
CPU	Intel	Xeon E3-1245V3 (BX80646E31245V3)	$289	Newegg
RAM	Kingston	32GB (4x8GB) 240-Pin DDR3 SDRAM ECC Registered DDR3 1600 Server Memory DRx4 Intel (KVR16R11D4K4/32I)	$347	Newegg
NIC	Intel	Networking Card Ethernet Server Adapter I340 Quad Port PCI Express Copper 1Gb (E1G44HTBLK)	$250	Newegg
Case	Cooler Master	Storm Stryker	$169.99	Newegg
Power Supply	===	===	$===	===
OS	FreeNAS	v9.1	$0!	===

Note: HBA Card or SAS expanders or anything like that is not necessary because the motherboard can already support 8 SATA3 drives, and I don't plan on having more than that in this tower.
**

HARD DRIVE	BRAND	MODEL	CAPACITY	RPM	CACHE	INTERFACE	QUANTITY
#1-4	HGST	Deskstar	4TB	7200	64MB	SATA 3	x4

TOTAL:

16TB

x4

Storage Server:

ITEM	BRAND	MODEL	PRICE	STORE
Motherboard	SuperMicro	===	$===	===
CPU	===	===	$===	===
RAM	===	===	$===	===
Graphics Card	===	===	$===	===
Case	Cooler Master	Storm Stryker	$169.99	Newegg
Power Supply	===	===	$===	===
OS	FreeNAS	v9.1	$0!	===
HBA card	IBM	m1015	$110	ServerSupply
SAS-to-SATA cable (SFF-8087)	(N/A)	1m, forward breakout cable	$10	Monoprice

*
**

HARD DRIVE	BRAND	MODEL	CAPACITY	RPM	CACHE	INTERFACE	QUANTITY
#1-4	Western Digital	Black Caviar	2TB	7200	32MB	SATA 3	x4
#5-6	HGST	Deskstar	2TB	7200	64MB	SATA 3	x2
#7-8	HGST	Deskstar	3TB	7200	64MB	SATA 3	x2
#9-12	HGST	Deskstar	4TB	7200	64MB	SATA 3	x4
#13	HGST	Deskstar	2TB	7200	32MB	SATA 2	x1

TOTAL:

36TB

x13

OK, folks, I know I've done this before and I'm doing it again.  But this time I think I'm REALLY going to do it.  I'm going to build my massive (and I do mean that literally, see below) server.  I've collected about a year or two of notes, and while I have not figured everything out yet, a little more back and forth here and I'll be ready to go.

First, here are the related threads so far:

https://www.donation...ex.php?topic=20801.0
https://www.donation...ex.php?topic=32359.0

And my first request that built my rock-solid desktop:
https://www.donation...ex.php?topic=16186.0

Eventually, I'll add a table of the parts and everything here.  The primary goal is to hold a lot of drives, over 20 of them.  The rest of the decisions will basically be built around that one.  I didn't want to go through the standard server rack route because the noise that results in cramming all those drives into a tight space.  The other thing I want to accomplish is that I want this to be the only computer in the house (other than laptops and tablets), so it will be in one room, and the rest of the house will be wired with ethernet/etc.  I haven't figured all that out yet, but that's the goal.  In the rooms, I just want monitors and keyboards/mice, but the computing is being done on this machine.

OK, here's what I have so far:
I'm pretty sure I'm going to go with the MASSIVE Caselabs TX10-V case ($800).  I like how people there say it will be the "last case they ever buy".  I like that.  I love the room it offers.

I can't decide on a motherboard yet.  I want a physically large motherboard with lots of PCI slots. But I also want as many USB 3.0 slots as possible, and esata if possible.  I know I have to figure out which of those SAS expander or whatever cards to get because of all the drives (I have lots of notes, just have to sort through them).  But I mention the USB 3.0 and esata because I like those and they seem to be rare.  Also, I probably don't have to worry about this, but recently I experienced problems transferring 4TB drives between computers because one of them required a "intel rapid storage technology" (I'm guessing it was before the large drives were common) and because I didn't have it installed, it sort of corrupted the drive, and I had to do a chkdsk and recover all that stuff.  I don't want to deal with that.  I don't want anything to happen to the drive when I plug it in to stuff.  I've experienced similar issues with a third-party USB 3.0 PCI controller I just got.  I don't like it, it worries me.
Please discuss!  Talking through all these things really helps me.

[wraith808]

What do you want the server for?  Other than to hold a lot of drives... there's several different applications that require a lot of space...

[40hz]

I love living vicariously through Superboy when it comes to a new build project!!!!

Even if there are limits to the vicarious experience

You might want to consider two servers? One which will have an i7 or dual Xeons (or dual i7s if they're out yet) if you're planning on running a lot of virtual machines. Then the other box would just be set up as a big storage server with a lot of drives. Separating them functionally shouldn't cost much more. And it will sure go a long way towards reducing heat and noise. And there might also be some savings to be realized since a storage server doesn't need the RAM and CPU power that an app server or VM  host should be given. There's cases built just for big NAS boxes that would probably be your best bet for that.

Good article on storage servers here. The biggest one they talk about holds ten drives. But the recommended case for that only runs $150 - so two are probably as inexpensive as getting one big one - and they will likely run cooler too with only 10 drives each. Plus you can always build another if you ever decide you need 30 drives someday.

There's also storage platforms from people like Addonics, Norco, Chenbro-Micom and others. And +1 with Wraith... let's refine the spec a little more before we start hardware shopping.

 

[Shades]

If the server OS needs to be Windows, go for Windows Server 2012. The management of the combined storage capacity from the hard disks you will be using, is very nice and rather simple. Just as Stoic Joker 

[Stoic Joker]

We just spun up an HP DL380p awhile back. As configured I've got 16 2.5" drives on two different RAID controllers in a 2U rack unit. Damn thing is insanely fast ... Full install from 1st POST to desktop took ~10min.

Did I mention the dual 8 core Xeons and 64GB of RAM?

eek! battery dieing...

[superboyac]

wraith:
I can't define a single function for it.  It will primarily be to hold tons of everything.  Videos take up the most space, I'm putting all my movies, all family recordings, etc. into it.  Many of them will be uncompressed, raw stuff.  They take up massive amounts of space.  But I'm also going to do everything else with it I normally do.  Virtual machines, tons and tons of software, content creation, games, arcade emulators, anything I can think of.  I guess I'm just a little tired of needing to build a machine every time i try a project.  I just want one ultimate one now, based around lots of hard drives.

40:
I've not been convinced to use NAS yet, mostly because it sounds to me to be a storage solution with a more restricted OS than full-scale Windows/Linux, and maybe just because I don't understand it or have less control (can't really tell).  I like the Norco units, no real problem there, that's an option.  I also like those ADdonics boxes, and when I was looking into those, the ones with the build quality I preferred were the ones from istardock (see here).  WHy?  They are made of metal vs. plastic.  I need to go through my notes to figure out which sas card expander thing I would need (I still don't really understand how to choose one, but i have a list of recommendations from here and around the web).  The two separate servers might be a good idea, I might even just use my current desktop (which has like a 7.8 windows performance rating  ) as the storage server, and the xeon/dual i7 thing as the high-performance vm box, what not.

Shades, I was thinking the same thing about Windows server 2012, I really want to give it a shot.  Especially the latest R2 version (preview version), because they supposedly significantly improved their drive pooling features.

Stoic...that sounds awesome, maybe I'll do something like that also.

[superboyac]

OK, so as a less insane/more practical option for the storage, how about this?

The only case out there I could find that is built for this sort of thing is the Addonics Storage IX case:
Please help superboyac build a server (2013 edition).

Then, I need the actual cages to stick in there.  My favorite ones, as I said, are from istardock:
Please help superboyac build a server (2013 edition).

So in the end, each box like this can hold up to fifteen drives.  I'll probably start out with one, then pretty soon move to two.  That should hold me over for a while  .

Remaining items for the high performance server:
motherboard
cpu (xeon?, dual i7?)
the card that will allow all the drives to plug in (sas expander?, raid?, i don't know the right term)

[tomos]

^FWIW, my small experience with 80x80 fans has been *very* noisy.
Maybe there are quiet ones out there?

[Stoic Joker]

Stoic...that sounds awesome, maybe I'll do something like that also.

-superboyac (July 31, 2013, 04:35 PM)

If you do keep a few things in mind:

• When this thing spins up it is loud as hell (8 Uber fans). We had to build it its own little room with a portable A/C unit ($280 at Lowes).
• Retail on this thing is north of $10,000 - But OMFG is it fast - Tehehe - I'm currently virtualizing Exchange 2007, SharePoint 2013, and 8 other servers on it ... And it ain't even blinked yet.
• The DL380p is a rack unit. Yes you can run it just fine sitting on a table...but it needs serious ventilation, and/or dedicated A/C unit.
• Lenovo also makes a nice Server with the same capacity (and similar price), that I'm running against this one to see who does best.
• Some sort of Hyper-V aware backup preferably featuring deduplication will be needed - Currently I'm up for suggestions on this point.

[Stoic Joker]

Forget about 3.5" drives ... The 2.5" are easier to pack in the case and cheaper to run.

[wraith808]

For what you said you are going to use it for... I'm with 40.  A NAS might be a better option.  With an OS comes other maintenance issues, and if you're not using it every day, but just accessing it, those can sneak up on you.  A NAS is meant for one thing... storage.  And with what you said, a single-responsibility type machine might be best.  It's really just a server that's optimized for serving files and such.

[Stoic Joker]

I'm not sure I'd want to deal with virtualizing anything on a NAS. But outside of that overkill probably should be kept to a minimum. I got the impression 40 was leaning towards two different boxes...each purposefully built to its specific task.

[wraith808]

I'm not sure I'd want to deal with virtualizing anything on a NAS. But outside of that overkill probably should be kept to a minimum. I got the impression 40 was leaning towards two different boxes...each purposefully built to its specific task.

-Stoic Joker (July 31, 2013, 06:07 PM)

It really depends on the NAS and the software.  A lot of them are OSes in and of themselves.

[Stoic Joker]

I'm not sure I'd want to deal with virtualizing anything on a NAS. But outside of that overkill probably should be kept to a minimum. I got the impression 40 was leaning towards two different boxes...each purposefully built to its specific task.

-Stoic Joker (July 31, 2013, 06:07 PM)

It really depends on the NAS and the software.  A lot of them are OSes in and of themselves.

-wraith808 (July 31, 2013, 08:40 PM)

Yes, for the storage aspect. But a NAS is just a big storage bucket that isn't going to have (a real rip tearing processor or a boat load of RAM) the resources to handle virtualization properly. They tend to be minimalistic and dutifully plod along much like a good mule. For virtualization you need cores lots of them...there is really no such thing as too many.

Which is why I was agreeing with 40 on the 2 server option, it'll keep the price down on the storage side because speed isn't a huge concern, and performance up on the virtualization side as you need a moderate amount of really fast stuff to keep it running properly.

We're currently reviewing a new potential client that hat a budget consciously built server that is tearing itself apart trying to keep 3 OSs running on storage class hardware. Their Exchange server (part of SBS11) is keeping the processor at 88-100% 24/7 ... But they're trying to run it on only 2 cores with 6GB of RAM (Yikes!). I gave our Exchange server 8 cores and 16GB of RAM and it's happy as a lark (However the accountants are not allowed in my server room.. ).

[40hz]

I'm not sure I'd want to deal with virtualizing anything on a NAS. But outside of that overkill probably should be kept to a minimum. I got the impression 40 was leaning towards two different boxes...each purposefully built to its specific task.

-Stoic Joker (July 31, 2013, 06:07 PM)

That's pretty much it. For a file server you'd want to keep it fairly focused with that capacity storage. I say 'NAS' in a generic way, but that doesn't mean it can't be running a 'full' OS.

I also try not to allow all my eggs to be placed in one basket unless it's a basket using 'enterprise grade' hardware. And even then I'd prefer it be on multiple devices.

Here's the thing... when you start stepping outside of the usual hardware framework and start considering special things like SATA port multipliers, you have enough driver and other concerns to not want to have to deal with it on an 'everything box.'

BTW: here's some good basic vids from Eli that are worth looking at if you're new to any of  
this:

 

[40hz]

Yes, for the storage aspect. But a NAS is just a big storage bucket that isn't going to have (a real rip tearing processor or a boat load of RAM) the resources to handle virtualization properly. They tend to be minimalistic and dutifully plod along much like a good mule.

-Stoic Joker (July 31, 2013, 10:18 PM)

Yep. To be honest, if I were planning some huge data storage system for my home use I'd be thinking of using a low power single board computer (there's a bunch of them) and setting up several file servers using them. A lot would depend on the storage the SBC and OS would be capable of efficiently handling, so that might take some research. But my inclination would be to just rack the drives on open shelving with plenty of airspace and one big quiet fan to keep everything cool.

So let's think outside the box for a second. Foregoing a closed case would dissipate a lot of heat and avoid any build up. And not having a fan restricted by the need to fit  in a case opens up a wealth of hardware options. (How about a couple of nice big Vornado table fans blowing on the drives instead of those miserable noisy little case fans?) The SBCs wouldn't need cooling or much power since they usually run off a wall wart power supply. And they could be mounted wherever it was most convenient to the drives they were attached to.

Hmm...gonna have to think about this ....

[superboyac]

Yes, for the storage aspect. But a NAS is just a big storage bucket that isn't going to have (a real rip tearing processor or a boat load of RAM) the resources to handle virtualization properly. They tend to be minimalistic and dutifully plod along much like a good mule.

-Stoic Joker (July 31, 2013, 10:18 PM)

Yep. To be honest, if I were planning some huge data storage system for my home use I'd be thinking of using a low power single board computer (there's a bunch of them) and setting up several file servers using them. A lot would depend on the storage the SBC and OS would be capable of efficiently handling, so that might take some research. But my inclination would be to just rack the drives on open shelving with plenty of airspace and one big quiet fan to keep everything cool.

So let's think outside the box for a second. Foregoing a closed case would dissipate a lot of heat and avoid any build up. And not having a fan restricted by the need to fit  in a case opens up a wealth of hardware options. (How about a couple of nice big Vornado table fans blowing on the drives instead of those miserable noisy little case fans?) The SBCs wouldn't need cooling or much power since they usually run off a wall wart power supply. And they could be mounted wherever it was most convenient to the drives they were attached to.

Hmm...gonna have to think about this ....

-40hz (July 31, 2013, 10:47 PM)

OK, now we're talking!  I really like the idea of the open shelf system because I would really prefer it to be quiet.  And I'll have a whole bedroom set aside for this sort of thing anyway.  I just need to figure out an easy way to put together a shelf system with the proper mounts and cable management, etc.  I could play around in sketchup to see if i come up with anything.  If anyone knows of any examples of this on the web, please link.

[40hz]

^Think about the layout. I'd suggest standing them vertically with an inch or so between. Heat also rises, so you'd ideally not want the banks of drives to be stacked one above the other.

Right now I'm looking at one of those old minimalist media racks (similar to pic below) and thinking hmm...



Something like that would need to be beefier...and thought given to vibration...definitely some sort of floating mounting would be a good since rigidity would only transmit vibration...but this could form the basis of the design. I also like the diagonal way they lean since that would make labeling easy to read. (Always indicate the date the drive was put into service on the drive itself. You'll thank yourself later when you start replacing drives.)

Alternatively, one of those kitchen rack or X-type dish things could be interesting from a design viewpoint. Some people have already used them as is for Raspberry Pi server projects.

Please help superboyac build a server (2013 edition).    Please help superboyac build a server (2013 edition).    Please help superboyac build a server (2013 edition).

Important to remember: maximum length for a high quality SATA cable is 2m for SATA-II/III - so that will have some bearing on the geometry of your design. And whenever in doubt, selecting shorter cables is usually the better choice.

Tom's has some discussion of throughput:

1. Device and mobo support SATA 2 - use of a SATA 3 cable will not get you the 6GB/s due to physical limitations of device and mobo.
2.Device and mobo support SATA 3 - use of a SATA 2 cable will not get you the 6GB/s due to physical limitation of the cable.
3. Device supports SATA 3, but mobo supports SATA 2 - using either SATA 2 or 3 cable will result in 3GB/s.
4. Device supports SATA 2, but mobo supports SATA 3 - using either cable will result in 3GB/s.
5. Device and mobo support SATA 3 - use of SATA 2 cable will result in 3GB/s
6. Device and mobo support SATA 3 - using a SATA 3 cable will get you the 6GB/s

...so as you can see, cable/drive/and host specs will also have an impact on total theoretical max throughput. Real-world will be slower due to OS and file system overhead.

Jeez...there's so much to think about.



---------------------

Addendum: for serious bespoke servers, I've always had the best luck with mobos and chassis from Supermicro. IIRC Stoic also had a recommendation for where to get used enterprise-grade 'big name' hardware that's well worth a look. I'm out of the office so I can't remember who they were or do a lookup right now. (I'm definitely gettin' old! )

[superboyac]

OK, that's a good start.  I'm a little uncomfortable without some kind of clamping in place or something.  I don't want a situation where the you accidentally bump into it and the drives move around or fall down.  I'm thinking maybe there are some toolless hard drive mounting kits, where I can place each drive in one, and then screw them in a custom wooden or metal frame with rubber stuff wherever is needed to reduce vibration and noise. Maybe something like this:

[40hz]

How about bungie-cording them to whatever using some hi-temp silicone tubing? That could serve double duty as a shock-mount and an anchor. Both ends, wrap over and under? I've seen similar done with servers being deployed for use in vans for 'mobile studio' and 'command center' applications.

[Stoic Joker]

While it does sound cool from a Dexter's Closet/mad scientist viewpoint. I'd doesn't really strike me as practical (i.e. How hard is it to move?). Just because the produced heat isn't confined...doesn't alter the fact that it is produced. Which is precisely why my office at home is constantly hot as hell.

The air will still need to be circulated.

The dust will indeed collect due to that extra circulation.

Things will start to get "fuzzy"..

When it comes time to move stuff for cleaning ... How sure of the structure will/can you really be?? HDD's are heavy and don't bounce for shit.

90% of the noise in the server room comes from all those tiny assed high speed fans. So for noise control just use a tower case, they have larger slower moving fans. Bulk of the drives can be the 7200 rpm Nearline SAS variety that aren't really noisy at all. This will make the mess much easier to control.

Remember the golden rule: The correct answer should involve the least administrative effort. If I don't keep the server's looking good...the wife will clean them for me ... Probably with soap...

So I make sure my cleaning task is as easy as possible.

[40hz]

Remember the golden rule: The correct answer should involve the least administrative effort.

-Stoic Joker (August 01, 2013, 11:58 AM)

Yes true. But this is his home server in it's own room. No real space constraints. And it's just him. Not 200 twits sneaking a look at their FB accounts while pretending to be working. He can do a shutdown and perform maintenance anytime he damn well pleases - no need to even send out that memo nobody ever bothers to read... (Uh, hi! This is Dale over in Marketing...is the server down? I have a report due and my boss wants to know when we'll be back up...)

The sysadmin's prayer.

Where it does have a big downside is that it's an experiment. No long-term QC or guarantee backing any of it.

But it's also just a storage server. So why not have some fun with it? The VM hosting machine is another story. For that you definitely want to go with known best practices.

[superboyac]

What's complicated about the bungee cord thing is that if you remove one drive, it affects the other drives on that twist.

What about a simple cage created with two rectangular plates, a couple of long threaded screws or rails to hold the drives, and then just rest the drives vertically in place, similar to the backblaze design, but with more standard home depot parts and more spacing between drives?  I did something like this in college.  what was nice about it is that it's very flexible in design, things are just bolted together so it's easy to take apart (no welding or anything).  A simple clamp of some sort, whether rubber or silicon, would hold the drives in place, should be sturdy.  attach the cables from the top.  basically the same as the dishwasher rack images above except it probably needs to be made from parts in the hardware store due to the custom spacing and clamping on the top, which i'd like to have.

then i get this bad boy to cool them:
Please help superboyac build a server (2013 edition).

[Stoic Joker]

Remember the golden rule: The correct answer should involve the least administrative effort.

-Stoic Joker (August 01, 2013, 11:58 AM)

Yes true. But this is his home server in it's own room. No real space constraints. And it's just him. Not 200 twits sneaking a look at their FB accounts while pretending to be working. He can do a shutdown and perform maintenance anytime he damn well pleases - no need to even send out that memo nobody ever bothers to read...

-40hz (August 01, 2013, 12:12 PM)

Yes, but home has its issues as well. One tends to be more relaxed about their movements and behavior. So delicate exposed stuff is not ideal. Someone sprays air freshener into the room from the doorway and the heavy wet mist settles..? An innocent casual gesture snag a cable or two. Or just the simple act of moving/cleaning the assembly spotlights a fatigue point...leaving a pile of drives on the floor. My kids are gone, so while for the most part my environment is perfectly safe. I do have grandchildren that could visit. We also have friends with children. Should I mention pets???

Point being: Never tempt fate...shit happens just fine on its own.

[superboyac]

Remember the golden rule: The correct answer should involve the least administrative effort.

-Stoic Joker (August 01, 2013, 11:58 AM)

Yes true. But this is his home server in it's own room. No real space constraints. And it's just him. Not 200 twits sneaking a look at their FB accounts while pretending to be working. He can do a shutdown and perform maintenance anytime he damn well pleases - no need to even send out that memo nobody ever bothers to read...

-40hz (August 01, 2013, 12:12 PM)

Yes, but home has its issues as well. One tends to be more relaxed about their movements and behavior. So delicate exposed stuff is not ideal. Someone sprays air freshener into the room from the doorway and the heavy wet mist settles..? An innocent casual gesture snag a cable or two. Or just the simple act of moving/cleaning the assembly spotlights a fatigue point...leaving a pile of drives on the floor. My kids are gone, so while for the most part my environment is perfectly safe. I do have grandchildren that could visit. We also have friends with children. Should I mention pets???

Point being: Never tempt fate...shit happens just fine on its own.

-Stoic Joker (August 01, 2013, 01:37 PM)

I live alone.  I have two rules for guests:
1) You can do whatever you want if you're invited.
2) Nobody is allowed to touch anything in the computer room/office without explicit permission.  Not family, not girlfriends, nobody.