Please help superboyac build a server (2013 edition).

[40hz]

Sorry for the confusion. In my case I'm used to seeing RAID included with port multipliers so I forgot a RAID card isn't the only way to do pooling.

RAID technology is not required to just do JBOD although many RAID cards also provide a JBOD option (i.e "drive spanning or pooling" which is not to be confused with RAID-0 "drive striping" although they are somewhat similar in concept).

Just to add to the confusion, have you considered using Gluster? It may be closer to what you want to do since this is a pure scale-out as opposed to scale-up approach. This approach will allow for virtually infinite expansion and flexibility.

Take a look at this:

General intro into the background and design of Gluster:

https://www.youtube.com/watch?v=eWIEA3f4y44

Intro to techt concepts (kinda long but very interesting IMHO):

https://www.youtube.com/watch?v=yhjEdDWo5jo

Installing a Glsuter server (short):

https://www.youtube.com/watch?v=pJEsB-gUdKk

Creating storage volumes on a Gluster server farm (short):

https://www.youtube.com/watch?v=dsgFHJlRURE

[superboyac]

(I've updated the first post with several components.)
Server experts: please check my motherboard, RAM, and CPU selections and let me know if I'm doing something crazy.  Are they compatible?  Is it too overkill?

40hz:
regarding GlusterFS (I have to finish watching all the links you posted later), what makes me nervous about it is that there isn't too much chatter about it online.  I've heard a lot of good things about ZFS and how reliable it is and proven.  There isn't talk like that about glusterfs yet.  The only bad thing about ZFS is that it's impossible to modify your storage pool architecture once it's set without re-doing it from scratch.  But other that that it is considered the most reliable storage system.  However, regarding that issue, it's not likely I'll be adding drives in and out all the time.  After thinking about it more, with multi-drive pools like this, if I do run out of space, it's probably better to just build another box.  I like this modular approach better now.  SO instead of having a 24 drive unit, I'll have multiple 4-8 drive units.  Easier to carry, easier to build, less stressful planning, and it also seems like a smarter and more reliable way to use groups of drives I have lying around.  Mixing them together in large pools seems to be a worse idea if you're not using enterprise SAS drives.

I was trying to figure out which OS/filesystem should I attempt first?  I was initially thinking Server 2012 R2 because that would be the least painful for me.  But like glusterFS, there doesn't seem to be much talk about it.  I'd rather start with FreeNAS because it seems like if I have issues, I can find the answer easily online.

[superboyac]

Looks like I have to be careful about my case selection.  Seems like Supermicro motherboards, even if it says ATX, may not truly be standard:

Supermicro Mobo's have always used proprietary solutions for their boards. For starters most are made to support 2u chassis, for that reason none of them are standard. Second even when they refer to E-ATX. There is no standard for it. Heck half the companies out there switch between calling it AL-ATX and E-ATX, the major issue and will always be an issue with Supermicro is they will always choose willy nilly the position of the connector panel in comparison to mounting points, and that there motherboard can extend higher then ATX above the top mounting point.

So unless you are getting an ATX workstation designed mobo from SM, don't buy a case till you have called the case manufacturer and checked with them on actual support for the particular SM mobo.

[Stoic Joker]

Seems like Supermicro motherboards, even if it says ATX, may not truly be standard:

hehe No ... But they are really nice. My server has been running on one for years. Their cases are quite nice also ... Very well made.

[superboyac]

Seems like Supermicro motherboards, even if it says ATX, may not truly be standard:

hehe No ... But they are really nice. My server has been running on one for years. Their cases are quite nice also ... Very well made.

Damn.  I have to contact them and get the dimensions.  I want it to fit in that cooler master case!

[40hz]

Seems like Supermicro motherboards, even if it says ATX, may not truly be standard:

hehe No ... But they are really nice. My server has been running on one for years. Their cases are quite nice also ... Very well made.

Damn.  I have to contact them and get the dimensions.  I want it to fit in that cooler master case!

The mobo measures 12" x 9.6", (30.48cm x 24.4 cm) according to Supermicro's spec page here.

Official standard dimension for an ATX mobo is 305mm x  244mm - so this Supermicro board falls almost perfectly within the ATX spec.

CoolerMaster says the case can support microATX, ATX and XL-ATX.

Since the XL-ATX form factor is 343 mm x 254 mm, it would appear that this case can accommodate a maximum of 305mm x 254mm using the largest dimensions of the eligible form factors. So I'm very confident the board you've spec'ed would fit the case you want to use.

------

re: OS choice:

In your case I'd do a feasibility test with Windows Server 2012 using the trial. If that works, you have a fallback. Only catch is you'll need one license per server - so it can get expensive down the road as you add servers.

If that checks out ok, next try a FOSS solution and see if you're comfortable and confident enough to want to use that instead.

FWIW, Microsoft's tech support is very good for all their server products. Very professional and quite different than the support offered for their desktop OS. There's several videos up on YouTube about stroage spaces under Windows Server that probably are worth a look.

For FOSS inspiration, check this vid out. Solid advice:

https://www.youtube.com/watch?v=ycbq_gTqT5M

[Vurbal]

FWIW, Microsoft's tech support is very good for all their server products. Very professional and quite different than the support offered for their desktop OS.

That matches my experience as well. Their server support team is very serious about making sure your problem gets fixed. In fact a Novell admin I used to know who managed systems for a number of companies (banks mostly) and really disliked MS servers had horror stories about Novell support and nothing but good things to say about dealing with Microsoft.

[superboyac]

The mobo measures 12" x 9.6", (30.48cm x 24.4 cm) according to Supermicro's spec page here.

Official standard dimension for an ATX mobo is 305mm x  244mm - so this Supermicro board falls almost perfectly within the ATX spec.

CoolerMaster says the case can support microATX, ATX and XL-ATX.

Since the XL-ATX form factor is 343 mm x 254 mm, it would appear that this case can accommodate a maximum of 305mm x 254mm using the largest dimensions of the eligible form factors. So I'm very confident the board you've spec'ed would fit the case you want to use.

------

Thanks 40.  That's good by me then, I'll get it soon when I'm ready.  I'm getting excited...I don't know why I love storage so much.  I must be like a virtual packrat or something.  I consider it like a second brain sort of.

[40hz]

I'm getting excited...I don't know why I love storage so much.  I must be like a virtual packrat or something.

Could be part of it.

But servers are cool tools. Period. I love 'em.

Having one of your very own opens up so many possibilities beyond just a desktop that it's a rush for most people.

[Vurbal]

I'm getting excited...I don't know why I love storage so much.  I must be like a virtual packrat or something.

Could be part of it.

But servers are cool tools. Period. I love 'em.

Having one of your very own opens up so many possibilities beyond just a desktop that it's a rush for most people.

Even an ancient server like my dual P3 Intel white box just gives off a different vibe than a desktop between the 10,000RPM SCSI drives that sound like miniature jet turbines when they spin up and the fans you can hear from the next room. OTOH it's damn annoying when I'm making a desktop capture video and have to shut it down to get rid of the background noise. 

I thought about putting a window air conditioner and wall mount server rack in the closet to contain it but the closet isn't deep enough.

[superboyac]

An in-depth article about Storage Spaces R2 and how to use it:
http://www.windowsnetworking.com/articles-tutorials/windows-server-2012/windows-server-2012-r2-storage-spaces-part1.html

Still, very few examples of people using this in real situations.  They are all articles talking about the new features, or this one where a guy is playing with it inside a VM (like me).  I can't find anything about people using this for real.  I also can't find how it compares to a ZFS system.  Remember, this is the R2 version, not the original Server 2012 version (or Windows 8 version).  I feel like freeNAS is going to be the way to go, but I will give this a shot since I'm just curious.

[superboyac]

For FOSS inspiration, check this vid out. Solid advice:

https://www.youtube.com/watch?v=ycbq_gTqT5M

Very nice video!  that's exactly what I'll be doing.
>100MB/s transfer speeds!  Bam!  That's what I'm talking about!  Looks like his server is running plain ol windows.  I know that's another possibility, probably the easiest.  But I still think I'm going to end up with freenas.  I've been reading a lot about zfs, and it sounds like that's the way to go.

I'm almost ready to order parts.  I need to know which ethernet cards to get and what other equipment I need to get for the ethernet stuff.  I'm guessing i will no longer be using my wifi router as the hub and will instead use whatever I stick into this box.

[superboyac]

Update:
added the NIC card as recommended from the streaming setup in the video above.  4 ports of gigabit, $250.
http://www.newegg.com/Product/Product.aspx?Item=N82E16833106050

[superboyac]

Well, ladies and gentlemen...the time has come! 

The first parts for the server has arrived.  I know, no big deal, i've built many computers.  but this is my FIRST server!  All the parts are pretty much the same, so i don't expect many surprises.  I'm more excited about the possibilities it offers, like centralizing some of my operations and stuff.  I'm being pulled in many directions these days, and I'm hoping the server will help.

[40hz]

^You are going to absolutely love having a server. I predict you'll soon wonder how you lived without one.

[superboyac]

OK, so far so good.  I built it, it starts up, no problem.  Now, I'm already being psychologically hit by the number of drives I need because of all the redundancies.  On one hand, i want to say screw the redundancies, but I learned my lesson on that 15 years ago when I lost some data I thought I'd never lose because of those damned deathstars from hitachi.  So I'm sticking to the plan!

So here's my experience with storage spaces so far.  With storage spaces (and probably other similar systems, this is my first time with them) there's a hierarchy to the storage organization.  it works like this:
bunch of physical disks --> storage pools --> virtual disks --> (volumes?)

I don't get the volumes, I don't know if I need them.  I guess I do, I'll find out.
My plan is to create two pools...one for the live, working files...and another pool strictly to backup the first one.  Within each pool, I'm going to use the mirroring option in storage spaces.  So this halves my drive capacity.  And since I'm backing the pool up, I'm essentially quartering my usable physical disk space.  Which is annoying since I'm now quadrupling my cost!!

So I initially thought I had plenty of space and drives, and I guess now I feel I'm quickly running out.  I think I mentally was prepared for one level of redundancy, not two.  Oh well.

i wasn't planning on getting an additional expansion card for more sata ports, but I may have to now.  The 8 ports on the motherboard are full.

[Shades]

Or you could buy 7 of those 8TByte drives (I am not sure if you can use those to boot from) and a 3TByte boot disk. That would leave you with 59TByte of storage space. That should cover your needs...for a while at least.

If that is not an option, you might consider to differentiate between data you want to keep and data you need direct access to. Direct accessible data is the data you need to store on your server, the other data could be on a NAS, a cloud, a set of (portable) hard disks or DEV/NULL...the only storage facility that dwarfs all the clouds combined!

Spoiler (click to show/hide)

Just don't expect to retrieve anything

[superboyac]

Or you could buy 7 of those 8TByte drives (I am not sure if you can use those to boot from) and a 3TByte boot disk. That would leave you with 59TByte of storage space. That should cover your needs...for a while at least.

If that is not an option, you might consider to differentiate between data you want to keep and data you need direct access to. Direct accessible data is the data you need to store on your server, the other data could be on a NAS, a cloud, a set of (portable) hard disks or DEV/NULL...the only storage facility that dwarfs all the clouds combined!

Yea, I think you are right.  One of the reasons for the server was to consolidate all the stuff I have lying around on spare drives, dvds, cds, external drives.  They're not terribly important, probably mostly disposable, but that was the plan.

I'm keeping an eye out on those huge hard drive sales.  Let me know if there's a good deal!

[superboyac]

OK, so I've now been able to test some things.
I'm running the storage spaces on mirrored mode, which is like raid5.  the other modes are simple (raid0) and parity (not recommended yet because it is so slow).

I was expecting to get speeds close to 100MB/s.  I am not.  I am getting somewhere around 50MB/s avg, and for transfers of a bunch of tiny files (pictures, tiny db files), it's even slower, maybe like 10-20MB/s avg.
If I just use an esata port on a normal computer (not this server), I can get about 80 MB/s avg, but that's also not mirrored or anything, just moving stuff to an external hard drive.

My immediate thought was to ditch this and instead use a ZFS setup, since there is a lot of evidence that zfs is fast and efficient with this kind of thing.  But, setting up the disks and pools with storage spaces was so nice and easy, I'd still like to give it a shot.  I also don't know if there are other tweaks I can try, and I don't know how much faster ZFS will be.  If I only get 10 MB/s increase in speed, I don't know if I care so much.

So that's where I'm at currently.  I'm debating storage spaces vs. ZFS.

[Shades]

When you add mirroring, your file-system will generate more overhead. Simply stated, you sacrifice speed for stability. Doesn't matter which file-system and/or RAID setup you use.

Also, verify the health of the disks in your storage pool(s). It is possible that one of your disks is generating a lot of interrupts, dragging the speed of all other drives down.

There is still a long shot (especially with older drives): drive alignment. Any partition management tool (Minitool has freeware for this) can verify this for you and allows you to align one or more partitions. If the partitions id filled with lots of data, this will take quite some time (hours). An empty partition is aligned within seconds. This really matters if disk I/O is of significance to you.

If you want speed, having a trustworthy backup strategy in place for your data and turning off any mirroring functionality is your best bet.

And if you are not comfortable with that, then your backup strategy isn't as trustworthy as you think

[4wd]

Just wondering, if your talking about a lot of files that rarely change, eg. a reasonably static file server or even some of the drives having static contents, would it be worth your while to use some form of "snapshot" RAID on them rather than a full blown RAID solution?

eg. SnapRAID, disParity - both free, or FlexRAID - not free.

Those are the ones that will work with existing filesystems, ie. you don't need to reformat to some other filesystem, (eg. ReiserFS, ZFS, etc).

That way you won't get the performance hit associated with a normal RAID system.

[superboyac]

Just wondering, if your talking about a lot of files that rarely change, eg. a reasonably static file server or even some of the drives having static contents, would it be worth your while to use some form of "snapshot" RAID on them rather than a full blown RAID solution?

eg. SnapRAID, disParity - both free, or FlexRAID - not free.

Those are the ones that will work with existing filesystems, ie. you don't need to reformat to some other filesystem, (eg. ReiserFS, ZFS, etc).

That way you won't get the performance hit associated with a normal RAID system.

I remember looking at these options.  After doing my test now, the one that appears most attractive is snapraid.  I'll give that a shot.

[4wd]

SnapRAID has a third party GUI, (Elucidate), that'll help with scheduling, syncing, etc but you'll still need to hit the CLI if you have to do a restore, (they haven't done that bit yet).

Re. SnapRAID, check your Volume Serial Numbers are all different, as per http://sourceforge.net/p/snapraid/discussion/1677233/thread/1e509458

[superboyac]

I changed my mind.  Now I'm going to try FlexRaid.  If I like it, I'll buy it.  if not, I'm going back to storage spaces.

I tried freenas last night.  It was too much for me.  I just don't have the time these days to learn a new interface or command line anything.

[4wd]

That was quick