Barebone server: what else do I need to complete it?

[superboyac]

I've been struggling with server lingo for several months now.  I've had many discussions with people and nobody seems to be able to clarify much for me.  Maybe I'm thickheaded.  Also, these discussions tend to turn into "Why the hell do you need something like this"...I'm just having a really hard time figuring out what I need.

So essentially, I want a server.  And I want to shove in 20 sata drives in there of varying sizes and makes.  I want to pool them in all sorts of flexible ways.  I don't want RAID because the drives are all different.  I have another thread about the drive pooling softwares I'm considering, specifically to address the hard drive issue.  I really like this Supermicro server:
http://www.provantage.com/supermicro-ssg-6047r-e1r24n~7SUP933V.htm?source=googleps

Now it says it's barebones.  What I really want to know is a list of additional parts I would need to plug this thing in and have it up and running.  So please help me with that.

It sounds to me like the hard drive slots are all there.  It also includes the xeon motherboard.  So is this what additional items I need to get?:
--RAM
--Xeon CPU

That's all I can see.  Sounds like everything else is already there like the network/ethernet switches and stuff.  And can I put any OS on there as I wish?

Currently, I have desktop computer connected to a standard wifi router and a modem.  All other laptops and tablets connect through the wifi router.  When I put the server in, I'd like the desktop directly connected to it through ethernet.  And everything else wifi.

So as it stands, this is what I need to purchase as separate items:
--supermicro super server
--xeon cpu
--some RAM

that's it?

[eleman]

Yup, buy the server, add the cpu, mix with some ram, leave in the oven for 20 minutes, you're ready to go.

Though this server has 10 sata interfaces, you won't be able to cram in 20 sata drives.

[superboyac]

Yup, buy the server, add the cpu, mix with some ram, leave in the oven for 20 minutes, you're ready to go.

Though this server has 10 sata interfaces, you won't be able to cram in 20 sata drives.

yeah, I was confused by that also.  What would I need to do to be able to use all 24 drives (SATA)?

[eleman]

Yup, buy the server, add the cpu, mix with some ram, leave in the oven for 20 minutes, you're ready to go.

Though this server has 10 sata interfaces, you won't be able to cram in 20 sata drives.

yeah, I was confused by that also.  What would I need to do to be able to use all 24 drives (SATA)?

There are pci-x expansion cards which provide additional sata ports. But I can't figure out how many pci-x1 slots this server has.

[superboyac]

I have this in my notes:

I would just run two SASLP cards and be done with it.  Tried and true, and it will probably end up being cheaper than using an expander anyway.  No bottlenecks either.

So to use all the drives, would I need two of these cards.  That would give me what they call 8x2=16 additional SAS slots, which I think means 16 more drives can plug into them.  So that gives me 16+(8+2)=26 drives, which covers all the slots.  Right?

questions:
Do I need to be concerned about the card's 3.0Gbps speed vs a few of my Sata3 drives which are 6.0Gpbs?
I can plug a PCIe card that is x4 into a slot that is x8 or x16, right?

Yup, buy the server, add the cpu, mix with some ram, leave in the oven for 20 minutes, you're ready to go.

Though this server has 10 sata interfaces, you won't be able to cram in 20 sata drives.

yeah, I was confused by that also.  What would I need to do to be able to use all 24 drives (SATA)?

(you were typing while I was responding)

There are pci-x expansion cards which provide additional sata ports. But I can't figure out how many pci-x1 slots this server has.

Looks like we're on the same page.  This server has:

4 (x16) PCI-E 3.0 slots
1 (x8) PCI-E 3.0 slot
1 (x4) PCI-E 3.0 (in x8) slot

[superboyac]

Looks like another issue is whether these SAS expander cards can handle 3TB and 4TB drives, which I have several of.  There are contradictory comments on newegg:

Dropped this in a new unRAID box and it worked right off. 5 2TB drives and one 4TB drive showed up without any issue.
The card came with the new firmware (.21) and was perfectly fine with my 4TB drive, have it mounted and in the array right now.

Comments were somewhat deceptive. the SAS2 version (SuperMicro AOC-SAS2LP-MV8) of this card can handle 3TB drives. That is not the card you are buying though. The drivers that are mentioned in the comments are for the SAS2 card, not this SAS1 card. If you intend to use 3TB drives look elsewhere.

[f0dder]

You don't need to worry about interface speed - harddrives are no near even 3gbps speeds (that's 300MB/s throughput), only high-end SSDs surpass that.

[TaoPhoenix]

Looks like another issue is whether these SAS expander cards can handle 3TB and 4TB drives, which I have several of.  

That's a lot of storage!  
And I was happy with my humble 1.75 TB from six years ago.

[40hz]

Take a look at how Backblaze has configured their 67-Tb servers to economically handle 45 drives. Article here which includes detailed parts list.

I'm not sure what "barebones" means since I've seen it applied to everything from a case+mobo+PS combo all the way up to a machine that just needed RAM, a HD, and an OS. So based on the list below, you'll need to get whatever isn't included in the barebones configuration you're buying.

What you need for any server is:

• Operating system (decide on this first because it affects everything else you use)
• RAM sufficient for whatever you plan on doing (rule of thumb: as much as you can afford/will fit)
• CPU - 64bit. Period. Speed and cores (or dual-CPUs) dependent on what it's for plus what OS you've selected.
• Power Supply - calc power requirements based on what's going in box and add about +20% over.
• Disk Drives - SATA. Period. (add expansion controller cards as needed - and watch power consumption!)
• DVD drive - to load OS and software. Cheap or recycled is ok here.
• Additional Cooling - the more in the box, the more air you'll need to move to keep everything happy. (Don't skip this.)
• NIC - two quality Gb-speed NICs minimum. Ideally with wake-on-lan features.
• Mobo - mostly determined by CPU selected. Get a server mobo - not one designed for desktops
• Case - determined by space/cooling requirements for all of above. (You can get really creative with this so think outside "the box.")

Next comes compatibility issues. You'll need to positively confirm the more esoteric elements in your configuration (expansion cards, BIOS, chipset, etc.) can all work with each other. So you'll need to check various manufacturer's hardware compatibility info and rework your spec as needed. That's one reason why having a vetted spec like the Backblaze server comes in so handy. They've already done the legwork and identified a "known good" configuration. (But it may not be well suited for what you have in mind.)

Configuring a server is an iterative process. You really can't say it's a go until you've successfully spec'ed all the components. It usually takes a few passes to get everything right.


re: Supermicro E5-2600 Series

I like Supermicro products. Unless they've changed things, barebones for them means the case plus whatever is soldered on the motherboard. So you'd need: CPU(s), RAM, disk drives, additional fans, power supply (and/or secondary PS), plus an OS.

Note: rackmount servers are heavy, noisy (as in 747 takeoff noisy) and run hot. So think about where you're gonna put it. Server rooms are usually soundproofed, and almost always have their own separate air conditioning, for a reason.

[superboyac]

Thanks 40, I was hoping to get your input.  I was so impressed by that backblaze project.  I'm really glad they've went out of their way to explain how they put everything together, I'll study it this weekend and try to mimic it on a smaller scale.

Until I move, I'm not sure what to do about the noise and heat.  I have a couple of spare rooms, but not the ability to cool them in a special way.  Would one Supermicro box really be all that loud and hot?  I have 6 drives connected to my desktop right now, and it's not that bad, and i don't think they get too hot.  I'd like to visit one in person to see really how bad it is.  I *may* be able to get away with a NY-style window mounted cooling unit in my complex, but I doubt it.

[mouser]

Note: rackmount servers are heavy, noisy (as in 747 takeoff noisy) and run hot. So think about where you're gonna put it.

ditto. 

[40hz]

Would one Supermicro box really be all that loud and hot?

Yup.

Any rack mount server is going to be noisy since they "turbo" the airflow. Small hi-volume fans going through a relatively small air passage. Whooosh! They're designed to accommodate server room density requirements - not ergonomics or aesthetics. And the heatsink fans on the CPUs are geared for performance and cooling - and "who cares!" about the noise levels.

Besides, anything with a bunch of disk drives in a relatively small case along with a couple of serious CPUs is going to be a very efficient space heater. The cases are designed to be heat radiators to cool the innards. Think of a rack mount server case as a giant heat sink for the system.

Thanks 40, I was hoping to get your input. 

You're welcome. But let's get Stoic Joker, JavaJones, Wraith808,  SteelAdept, skwire, 4wd, Edvard, f0dder, and some of the other people who are involved with servers (sorry to the other DoCo members whose names I can't recall off the top of my head - you guys know who you are! ) in on this too.

[Stoic Joker]

Note: rackmount servers are heavy, noisy (as in 747 takeoff noisy) and run hot. So think about where you're gonna put it. Server rooms

+1 x 10 I just got a great deal on an off lease Dell (1U rack mount) data center server. On start up that thing will flat-out wake the freaking dead! - (Think locked it a closet with 19 vacuum cleaners) - Hell, even with the fans at 'idle' it gave me a headach after sitting next to it for a few hours.

@SB Supermicro does make some great stuff. My home office's main server is a supermicro box that has been running rock steady 24/7 for about 6/7 years now. Unless you have a basement you're not using ...(seriously it really is that loud)... Go for a tower server with an external array.

[40hz]

Go for a tower server with an external array.

Agree completely.

Also, housing the disk array in a separate enclosure will go a long way to distributing heat and keeping as much of it as possible away from your $$$ Xeons. (Looks sexier too IMHO. Racks look like meh. Two nice towers with all those blinkin' lights look ever so much cooler. (Run cooler too!)

Additional note: if your board supports it (and virtually all "server" mobos do) I would spend the extra to get ECC RAM. Especially if I was going to be provisioning for virtual machines, software RAID, or drive pooling. One less opportunity for unexpected surprises cropping up with ECC.

Many memory suppliers discourage people from using ECC since it will reduce performance marginally (1-2%) compared to non-ECC memory modules. But they're primarily addressing the desktop/workstation environment where (with current RAM products) it's not considered needed or desirable.

Servers, however, are a "whole 'nuther smoke." They're not just a tower PC loaded with drives and maxed out with RAM. With real servers, bullet-proof reliability, redundancy, and low-latency data bus engineering are more important than wringing the last fly speck of performance out of each of your individual components. You have to see them as 'one thing' rather than a collection of components. And they really are designed to be "set & forget" devices. You power them up - and "that's that" if you did it right.

I've had servers run continuously for years on end. The only time they are were ever powered down was to add or replace a drive if they didn't support hot-swapping. Or (if they were Windows servers) to perform a required reboot following a software upgrade. I've routinely had BSD/Linux servers go for well over three years without requiring a single reboot. I have one that's been rebooted only twice in seven years and is still going strong.

FWIW there's some disagreement in the "pro" community about whether or not ECC is "worth it" any more. It's split about 60-40 against last I paid attention to it. And it seems to be largely an age dependent thing. The young 'uns say we older guys aren't up enough on advances in memory engineering. We codgers (i.e. anyone over 40) say these youngsters are far too trusting when it comes to reliability claims; and haven't been around long enough to see all the weirdness that goes down in a server environment to know.

The real truth probably falls somewhere in the middle.

So...I'm still inclined to spec ECC for a server. But I'm an "old guy." So maybe some other people might want to put their tuppence in in this one?

About ECC

[attach]

[f0dder]

So...I'm still inclined to spec ECC for a server. But I'm an "old guy." So maybe some other people might want to put their tuppence in in this one?

Even with "advances in memory engineering", what about stuff like... cosmic radiation? Even if you only get a single-bit corruption every few years, a single-bit corruption inside a compressed datastream can ruin quite a lot.

I'm about to shop for a new home server, probably in October, and considering whether to go for ECC memory... but that does mean also going for a server motherboard and Xeon CPU, and that does end up in a somewhat other price class than a commodity i3 or i5.

[40hz]

considering whether to go for ECC memory... but that does mean also going for a server motherboard and Xeon CPU, and that does end up in a somewhat other price class than a commodity i3 or i5.

Been awhile since I last looked, but Intel always used to have at least one non-Xeon server mobo that could use ECC. An i3 with ECC would be awesome for a home server.

Ok checking....and yup! They have one: Intel® Intel Server Board BBS1200KPR (not to be confused with similar earlier version #DBS1200KPR which doesn't support current Xeon versions)

Has an LGA1155 socket (supports Core i3 and Xeon E3-1200 series) and 2 DIMM slots (max 16Gb) for DDR3/SDRAM - either ECC or non-ECC, unbuffered. Integrated graphics. One expansion slot (ok...so this one won't work for Superboy's specification.) Sells for around $150 USD too...

Full tech specs (PDF-66 pages!) download here.

Nice board. Oooo look! It even comes in 10-packs just in case you're a real sport!

[Stoic Joker]

The real truth probably falls somewhere in the middle.

hehe Doesn't it always...

So...I'm still inclined to spec ECC for a server. But I'm an "old guy." So maybe some other people might want to put their tuppence in in this one?

I was just going to vote 'Old Guy' and shutup ...(because I'm "pushing Fifty")... But I figured what the hell...I'll tell on myself. Whle I too usually go for ECC, I have skipped it on a few occasions (to cut project costs) where I was dealing with a really small operation with 10 or less client machines, minimal processing requirements, and no reall need for planning to scale up in the near future.

[40hz]

^I too have reluctantly skipped ECC where cost was an issue for a "nickel & diming" client on a non-commerce webserver or similar "crap app" (kidding...just kidding!) and where a simple reboot cures most ills with no side-effect.

Today I usually just spec "server memory," not mention ECC, and get on with it. And if it ever does become an issue, I'll spec for non-ECC, tack on an extra half hour of time - and get the bloody ECC anyway. It's not that much more expensive. ($75 for a quality 8Gb module/$170 for 24Gb 3x8 kit vs $49 for a good quality 8Gb non-ECC desktop DDR3 module?) C'mon people, this is for a freekin' server - lets get real shall we?

[superboyac]

Thanks everyone, this discussion is massively helpful.  Keep going!

[Stoic Joker]

C'mon people[/i], this is for a freekin' server - lets get real shall we?

Understood. But I continuously end up walking into a place - because tragedy struck" (oh surprise...Not!) - with all of their mission critical data on a "Working" server running a client OS...that they bought at BestBuy 5+ years ago.

I try not to scream ... But I die inside, just a little, each time it happens. 

[40hz]

I try not to scream ... But I die inside, just a little, each time it happens.

Oh, I do too! But then I take comfort in the sure knowledge that they will learn a valuable lesson and gain a great deal of understanding about why it's not good to disregard proven advice and best practices when running an internal data center. A lesson that gets further reinforced when they receive the bill for the services required to get them back up and finally running correctly.

It used to bother me a lot. But not quite so much any more. I'm learning (albeit slowly) to not waste excessive sympathy on people who can't be bothered to learn about things critical to their businesses. Or run them in a responsible manner.

Personal computer users and small Mom & Pop operations I can feel bad for. (And I almost always cut them some slack and give them a huge break when it comes to billing. At least the first few times it happens.) But I have zero sympathy for bigger SMBs that are making money and should know better. Especially when their top management is ten years younger than I am, and should therefor be somewhat knowledgeable about this technology. (I've often seen far more managerial attention paid to leasing a photocopier, or buying a turnkey phone system, than was ever given to setting up a mission critical internal data network.)

If I feel bad for anybody, it's the rank & file employees whose jobs might be on the line as the result of a company's system crashing too hard. And I have seen a business tank shortly after its "summer intern-built server" (kept in an overheated storage closet) went mobo up - taking their general ledger and AR system with it.

When it comes to IT, management gets what it asks for. Good thing for them they don't (too often) also get what they deserve.

[superboyac]

I do like the idea of going with towers.  I initially wanted to do it this way, but where I ran into issues was finding a tower box that could hold more than 8 drives.  So then the question is, if a tower is holding 10-15 drives, why would that be any less heat or noise than the aforementioned rack server?  Whatever, I trust you guys on that.

I was also glad to see the backblaze guys using some addonics products.  i could never tell if addonics made quality equipment or not, so that probably means they are good.  Also, Addonics makes configurations where you can put 15 drives in a single tower if you use their customized 5-drive banks.  I like that.  I'll work on a setup and post it here, see if it makes sense.

[40hz]

So then the question is, if a tower is holding 10-15 drives, why would that be any less heat or noise than the aforementioned rack server?

It can be as loud. But it doesn't have to be. You're not cramming as much into as small a space - and you can take noise into consideration. Something the rack case designers definitely do not.

The other problem with rackmount is the form factor. Out of the rack, they're the size of a mutant pizza box -as this snapshot of a 1U server from one of "Levi the Computer Guy's" videos shows:

[attachthumb=#]

They look pretty svelte in the rack. But out of it they're something like 17"x28." And once loaded, they're unwieldy and drop prone if you're not expecting the weight or really weird balance. I'm no wimp when it comes to lifting things. But the first time I pulled a 3U ProLiant server completely out of it's rack, I almost lost it. And that puppy only had 6 drives in it.

One thing that complicates your design spec is needing to have removable drive caddies because you anticipate so much swapping in and out. If you get away from that slightly, the Lian Li "A" series has been a popular go-to choice for multi-drive server configurations. These cases are specifically designed for this purpose so they've engineered for adequate cooling and airflow. The Lian Li PC-A70F and similar are good choices. That could give you 3 externally accessible slots for tray mounted drive bays along with seven internal bays for the more permanent drives.

Lian Li makes a nice case. But they're expensive.

Something you might want to consider since you're an engineer...maybe design your own and have a sheet metal shop fab it for you?
If you're not doing a data center, why bother cramming stuff into the smallest case possible? You're only going to have one of these, right? (Right? )

I always though a case with staggered and spaced drives with large slow fans on top and bottom would provide vastly better cooling and less noise than traditional drive array cases. Possibly even metal grill (less heat build-up) if it didn't dissipate the airflow too much before it passed over all the drives...

Something like this very rough sketch:

[attach]

[f0dder]

I once joked I'd like to use one of those Dysan bladeless fans for it.

Interesting idea - Dyson makes some pretty awesome stuff. The first time I saw one of those bladeless blowers, it was almost indistinguishable from magic :-)

PS, SuperBoy... check YouTube for stuff like "my home server room", and pay attention to the noise in those videos.

[40hz]

check YouTube for stuff like "my home server room", and pay attention to the noise in those videos.

+1! There's one where some guy has this absolutely beautiful setup in what looks like his garage. According to the vid this is just the first stage of his planned installation. (He's a small private ISP.) You'd think you were sitting on an airport runway with the amount of wind noise in there.