topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Thursday March 28, 2024, 1:46 pm
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: Please help superboyac build a server (2013 edition).  (Read 78350 times)

Shades

  • Member
  • Joined in 2006
  • **
  • Posts: 2,922
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #100 on: August 07, 2013, 08:52 PM »
^ +1

So you are one from the (old) school of blood and magick as well... 8)

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #101 on: August 07, 2013, 09:10 PM »
And now they have plastic cases without the sharp edges... so what are you going to do?  The reason that I liked metal cases is because it seemed that every time I cut myself, things would start working.  It just likes blood, you know?  Like voodoo.

Or... is that a sign of psychosis?  ;D

Nope. The blood and the pain (and the risk of a tetanus infection) jogs some little grey cells and that one obvious beginner thing you thought you checked 200 tries earlier (but actually didn't) turns out to be the thing that needed doing.

bofh2.jpg

Ten minutes plus one reboot later and you're laughing grimly to yourself while hoping nobody ever finds out about your stupid oversight. (Especially if you plan on billing them for your actual hours.) Which is yet another reason why so many sysadmins prefer to work well after regular business hours - and preferably as close to 1:00am as possible. (Even the security guards are getting sleepy and coasting by then.) We're not nighthawks by accident. It's more like it's by design. THE BOFH STILL RULEs!

Uh-oh. I said that out loud didn't I?

I just *hate* it when that happens.

not-enough-coffee.png

 8)


Vurbal

  • Supporting Member
  • Joined in 2012
  • **
  • Posts: 653
  • Mostly harmless
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #102 on: August 07, 2013, 09:16 PM »
^ +1

So you are one from the (old) school of blood and magick as well... 8)
You say that like there's another way...  :tellme:
I learned to say the pledge of allegiance
Before they beat me bloody down at the station
They haven't got a word out of me since
I got a billion years probation
- The MC5

Follow the path of the unsafe, independent thinker. Expose your ideas to the danger of controversy. Speak your mind and fear less the label of ''crackpot'' than the stigma of conformity.
- Thomas J. Watson, Sr

It's not rocket surgery.
- Me


I recommend reading through my Bio before responding to any of my posts. It could save both of us a lot of time and frustration.

Vurbal

  • Supporting Member
  • Joined in 2012
  • **
  • Posts: 653
  • Mostly harmless
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #103 on: August 08, 2013, 03:10 AM »
Nope. The blood and the pain (and the risk of a tetanus infection) jogs some little grey cells and that one obvious beginner thing you thought you checked 200 tries earlier (but actually didn't) turns out to be the thing that needed doing.
I'm certainly not going to deny that's it some of the time. We all know I'd be lying.

But, at least for me, more often than not it was things like the fact IBM buried the compatibility notes for the ServerRAID III adapter in an obscure corner of their website and didn't bother to either include them in their server docs so I would know why the 8MB of free space that should have been at the end was at the beginning where the partition should have started. Nothing more fun than spending 6 hours beating your head against a wall only to solve the problem with a Win98 boot disk.  >:(

No, I'm not bitter. Why do you ask?

THE BOFH STILL RULEs!

Hail to the Bastard!
I learned to say the pledge of allegiance
Before they beat me bloody down at the station
They haven't got a word out of me since
I got a billion years probation
- The MC5

Follow the path of the unsafe, independent thinker. Expose your ideas to the danger of controversy. Speak your mind and fear less the label of ''crackpot'' than the stigma of conformity.
- Thomas J. Watson, Sr

It's not rocket surgery.
- Me


I recommend reading through my Bio before responding to any of my posts. It could save both of us a lot of time and frustration.

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,646
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #104 on: August 08, 2013, 06:46 AM »
THE BOFH STILL RULEs!

Hail to the Bastard!

So a user asks if they can have full access to the new $30,000 business management system that I'm in the middle of deploying. This new system will (er... should) combine our current menagerie of discontiguous systems into one easily accessible "roof" if you will.

But said user wants to be hotshot and go exploring... No.

My response?

bofhlogo.png

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #105 on: August 12, 2013, 09:50 AM »
I'm trying to figure out which HBA card I'm going to get to connect all these drives.  From what I can understand, people are saying it is not a good idea to use sas expanders with sata drives with ZFS.  SO that means I have to use non-sas cards.  The one that is recommend from here is the IBM m1015.  This was also the same card that someone here recommended to me, so maybe that's the way to go.

But what i don't get is that the m1015 seems to use sas expanders also.  No?  I don't understand.

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #106 on: August 12, 2013, 09:54 AM »
Of course, the other option is to just get real deal SAS drives.  It will cost me, though.

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #107 on: August 12, 2013, 10:56 AM »
So if I go the SATA route (no SAS expanders), I need an HBA and some port multipliers.  I don't know what HBA to get, but Addonics has some port multipliers:
AD5HPMSXA_diagramB.jpgPlease help superboyac build a server (2013 edition).

So maybe the IBM m1015 plus these multipliers?

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #108 on: August 12, 2013, 11:57 AM »
Regarding using SATA drives with SAS, here's some more input that is making me go round and round:
Most of what you've read about using SAS expanders with SATA drives is just plain old fashioned FUD - stuff posted by people repeating things they didn't really understand or stuff promulgated by one particular system vendor with an agenda.

There is no significant issue. There was a bug in Opensolaris-derived systems that got (incorrectly) blamed on the combination of SAS expander with SATA drives because systems in that configuration could be shown to exercise the bug - but it was clearly and unabiguously a bug in the OpenSolaris code and NOT a side-effect of using SAS expanders with SATA drives. The bug existed in general and all ZFS-based systems were actually at some risk from it whether they used SAS expanders or not. It was fixed in Solaris 11 and separately on other related branches (like OpenIndiana). All of the posted FUD derives from this. It was propagated in some self-serving posts by one systems vendor who either couldn't or didn't want to bother fixing the (well published) bug in their OpenSolaris-based branch.

But if it makes you uncomfortable using then then don't use them.

Besides, as apnar posted above, serving the typical SOHO max of 24 drives using 3 8-port HBAs is almost always cheaper and faster than 1 HBA + expander. But if you need to support more drives or if you don't have the PCIe lanes to spare for the extra HBAs then expanders work great too.
I really am trying not to have to go to SAS drives, because that would just cost me quite a bit (>$3000).  I want to use the SATA drives I already have.  If using SATA drives with SAS is not as big a deal as I'm reading about, maybe the best way to do it is:
--server motherboard (SuperMicro)
--IBM m1015 HBA card
--SAS expanders to SATA drives

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #109 on: August 12, 2013, 11:59 AM »
Is there a difference between a SAS expander and a breakout cable?

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #110 on: August 12, 2013, 04:13 PM »
Regarding the cables that connect the m1015 SAS ports to the actual hard drives, I came across this helpful post:
Hi electricd7,

I'm actually going to be bolting pretty much that exact motherboard-processor-memory combination into my existing filer tonight!

What you need are SFF-8087 cables. They come in 2 different "flavors":

Forward Breakout = crossover = SFF-8087 (Controller card) to 4x SAS/SATA ports on DRIVES
Reverse Breakout = 1:1 = SATA/SAS Ports (on Motherboard or controller) to SFF-8087 backplane

You will need "forward" cables available here:

http://www.monoprice...54&cs_id=1025406

I use a Fractal Designs R3 case and bought the .75M ones which turned out to be just about the perfect length to go from the card, down to the case bottom & back up to the 8 drive sleds.

While you are ordering the 8087 cables you might as well get some Molex-to-SATA converters as well:

http://www.monoprice...26&cs_id=1022604

Use these so you only need to get 2 Molex power connectors to the drives instead of 6 proper SATA power connectors, makes for a real clean install.

-Will


superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #112 on: August 12, 2013, 05:34 PM »
A guide for newbies to FreeNAS, updated with v9.1 info.  Nice!

Some of my personal highlights:
Please recognize that your Windows hardware knowledge may provide some small insight for selecting hardware but is not equivalent to expertly choosing hardware for a FreeBSD based system.  For example, ECC RAM in a desktop isn’t too useful.  But for ZFS it can be the difference between saving your data and a complete loss of the zpool with no chance for recovery.  Realtek NICs are common in the Windows world, but perform extremely poorly in FreeBSD.
FreeBSD has a very steep learning curve.  It is not for those looking to learn it in a weekend.  I was operating a nuclear reactor before I was old enough to drink alcohol, but I still spent a solid month getting familiar with FreeNAS.  Everyone’s mileage will vary and your level of “comfort” with FreeNAS will be different than mine.  But I haven’t lost any data and I have helped many people recover an unmountable zpool.
VDevs with single disks are known as “striped” disks. They have no redundancy.
VDevs can provide redundancy from individual hard disk failure inside the same VDev.
VDevs cannot operate outside of a zpool.
You cannot add more hard drives to a VDev once it is created.*
When a VDev can no longer provide 100% of its data using checksums or mirrors, the VDev will fail.
If any VDev in a zpool is failed, you will lose the entire zpool with no chance of partial recovery.
You can think of it simply as:
Hard drive(s) goes inside VDevs.
Vdevs go inside zpools.
Zpools store your data.
Disk failure isn’t the concern with ZFS.  Vdev failure is!  Keep the VDevs healthy and your data is safe.
ZIL drive performance will need to exceed the zpool performance of the expected workload to be useful.  Typically an SSD is used for this application.  An Enterprise class SSD or SSD based on SLC memory is recommended.
For maximum performance and reliability, you should never try to use ZFS with less than 8GB of RAM and a 64-bit system.
Intel Network cards are the NIC of choice.  The drivers are well maintained and provide excellent performance(not to mention inexpensive).  Other NICs have been known to perform intermittently, poorly, or not at all.  Realtek NICs can perform decently as long as you have a CPU that has enough power to process all of the network traffic.  (This is one thing that is VERY different between Windows and FreeBSD).  Using “low power” CPUs such as Intel Atoms and AMD C-70s are NOT powerful enough to be used with Realtek and get good performance.
ZFS has very few “recovery tools” unlike many other file systems.  For this reason, backups are very important.  If the zpool becomes unmountable and cannot be repaired there are no easy software tools or reasonably priced recovery specialists you can use to recover your data.  This is because ZFS is enterprise-class software, and no enterprise would waste their time with recovery tools or data recovery specialists.  They would simply recover from a known good backup or mirror server.
OK...this is the freaking tutorial I've been dying for!  :up: :up: Great stuff.  I'm only about 75% of the way through, but I'm tired now.
« Last Edit: August 12, 2013, 05:51 PM by superboyac »

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #113 on: August 13, 2013, 12:20 PM »
Call me crazy, but just for backup, I'm considering getting a standard plug-and-play NAS box.  Maybe from Qnap...a 4 or 5 bay one.  If it becomes redundant later after i have my custom boxes working, i can put them in a relative's home for off-sute backup purposes, etc.

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #114 on: August 13, 2013, 01:02 PM »
OK, yes.  I believe I am first going to do a small 4-drive test build.  I'm going to build a small-case NAS for 4 drives and that will be my first experiment.

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 11,186
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #115 on: August 13, 2013, 03:35 PM »
Call me crazy, but just for backup, I'm considering getting a standard plug-and-play NAS box.  Maybe from Qnap...a 4 or 5 bay one.  If it becomes redundant later after i have my custom boxes working, i can put them in a relative's home for off-sute backup purposes, etc.

Nope... I call you coming to your senses :)  And I'd recommend taking a look at a Synology Diskstation- I'm very satisfied with mine, and in the end, it made me realize I didn't really need a server anymore.

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #116 on: August 13, 2013, 03:47 PM »
Rosewill does some nice affordable bare-bone external units. So does SANS Digital.

I would definitely test what you're planning with a smaller incarnation before you committed whole hog to a big installation.

Might also want to check some of the forums (FreeNAS etc.) to see if anybody else has done something like what you're planning. Or tried and failed. Reason I'm saying is that there is a good chance there's a breaking point somewhere once you go over a certain storage capacity and start running into reliability issues with off the shelf open software.

Maybe somebody knows for sure one way or the other? 8)

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #117 on: August 13, 2013, 04:55 PM »
Call me crazy, but just for backup, I'm considering getting a standard plug-and-play NAS box.  Maybe from Qnap...a 4 or 5 bay one.  If it becomes redundant later after i have my custom boxes working, i can put them in a relative's home for off-sute backup purposes, etc.

Nope... I call you coming to your senses :)  And I'd recommend taking a look at a Synology Diskstation- I'm very satisfied with mine, and in the end, it made me realize I didn't really need a server anymore.
I'm going to have to intentionally ignore this post for now  :D. (but duly noted)
 FYI, I was just checking out the diskstation and qnaps versions also.

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #118 on: August 13, 2013, 05:34 PM »
Rosewill does some nice affordable bare-bone external units. So does SANS Digital.

I would definitely test what you're planning with a smaller incarnation before you committed whole hog to a big installation.

Might also want to check some of the forums (FreeNAS etc.) to see if anybody else has done something like what you're planning. Or tried and failed. Reason I'm saying is that there is a good chance there's a breaking point somewhere once you go over a certain storage capacity and start running into reliability issues with off the shelf open software.

Maybe somebody knows for sure one way or the other? 8)

From everything I've read so far, the thing that is sticking out the most for me is the hard drives.  I have a bunch of consumer grade drives that i basically got on sale the past year or two.  But they are not ideal for a freenas server, although most say that it works fine.  Really, the best way to go is SAS drives.  So based on this, I'm going to try a 4-bay smaller implementation of what i originally planned and test it a little bit before going huge and deciding to get sas drives or something.

There are a lot of hobby builds out there in the htpc community like xbmc folk and stuff.  So many of them use the Norco 24 drive racks, so they are dealing with tons of storage.  it sounds like if I'm careful and follow the guy's advice in that freenas guide above, I should be ok, but it will take some studying.

Another easier option I'm considering toying with is the original "soft" solution.  That is, using a normal windows 7 OS or windows server, and using the software tools to deal with the drive pooling like snapraid.  The advantage there supposedly is an easier and more flexible setup, like, I can pull drives out with less headache and add drives, etc.  The disadvantage is that there will likely be some performance hit.  And maybe a reliability hit also, i mean, there must be a reason why the enterprise users don't do that kind of stuff.

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,646
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #119 on: August 14, 2013, 07:00 AM »
i mean, there must be a reason why the enterprise users don't do that kind of stuff.

(Short answer> Battery on the RAID controller allows the cache to be written to (array) drives safely in the event of a power failure.

(However> MS was touting the performance/convenience of their new Server 2012/Hyper-V soft RAID/disk pooling feature for enterprise class deployments at the last show I went to.

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #120 on: August 14, 2013, 07:22 AM »
+1 w/Stoic on the battery on the card point. Do not use a RAID card that doesn't have a battery. If it was a toss up between a cheap SATA RAID card, or software RAID under Linux or Windows, I'd definitely go with software RAID and forget the card. At least for basic mirroring. Never tried RAID5 under software although I'm told by others that it's been reliable. There is a performance hit. But on a single user server (hmm...now that's an odd phrase) it shouldn't be a significant factor. However, I would still be concerned about the number of drives you try to incorporate under soft RAID. I'd worry with anything over three - and definitely not go past five drives in that scenario.

Note: Microsoft will give you a free 180-day trial copy of Windows Server 2012. Info and downloads here. I'd definitely give that a try, if for nothing other than to have something to compare should you decide to try a FOSS solution too.
 8)

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #121 on: August 14, 2013, 08:58 AM »
I do want to try the WIndows Server 2012 pooling features.  Maybe I'll do that first, and if it's a problem I'll go to freenas next.  Here's an article talking about the new R2 drive pooling features:
spaces-verus-arrays.pngPlease help superboyac build a server (2013 edition).

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #122 on: August 14, 2013, 09:25 AM »
If I go with Windows Server 2012 and their Storage Spaces, here are my notes from that:
http://technet.micro...ibrary/jj822938.aspx
Mirror
Stores two or three copies of the data across the set of physical disks.
Increases reliability, but reduces capacity. Duplication occurs with every write. A mirror space also stripes the data across multiple physical drives.
Greater data throughput than parity, and lower access latency.
Uses dirty region tracking (DRT) to track modifications to the disks in the pool. When the system resumes from an unplanned shutdown and the spaces are brought back online, DRT makes disks in the pool consistent with each other.

Requires at least two physical disks to protect from single disk failure.
Requires at least five physical disks to protect from two simultaneous disk failures.

Use for most deployments. For example, mirror spaces are suited for a general-purpose file share or a virtual hard disk (VHD) library.
There are three options: mirror, parity, simple.  So they are recommending mirror for my setup.

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,646
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #123 on: August 14, 2013, 12:03 PM »
Never tried RAID5 under software although I'm told by others that it's been reliable. There is a performance hit. But on a single user server (hmm...now that's an odd phrase) it shouldn't be a significant factor. However, I would still be concerned about the number of drives you try to incorporate under soft RAID. I'd worry with anything over three - and definitely not go past five drives in that scenario.


MB integrated Software 3 disk RAID 5 here ... Anything SB needs to know about it is contained in the (I think 4 page) thread here about when I goofed replacing a blown drive with the wrong type.

The third and final disk from that array blew just last week and replacement (with the spare I had on hand) went quite smoothly.

+1 to the 3 disk soft RAID max.



If I go with Windows Server 2012 and their Storage Spaces, here are my notes from that:
http://technet.micro...ibrary/jj822938.aspx
Mirror
Stores two or three copies of the data across the set of physical disks.
Increases reliability, but reduces capacity. Duplication occurs with every write. A mirror space also stripes the data across multiple physical drives.
Greater data throughput than parity, and lower access latency.
Uses dirty region tracking (DRT) to track modifications to the disks in the pool. When the system resumes from an unplanned shutdown and the spaces are brought back online, DRT makes disks in the pool consistent with each other.

Requires at least two physical disks to protect from single disk failure.
Requires at least five physical disks to protect from two simultaneous disk failures.

Use for most deployments. For example, mirror spaces are suited for a general-purpose file share or a virtual hard disk (VHD) library.
There are three options: mirror, parity, simple.  So they are recommending mirror for my setup.


??? Mirroring for a VHD library? Library meaning storing them only, or running them from that location? Mirroring cuts spindle count in half which will severely limit the number of VM's you can comfortably run simultaneous.

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Re: Please help superboyac build a server (2013 edition).
« Reply #124 on: August 14, 2013, 12:49 PM »
??? Mirroring for a VHD library? Library meaning storing them only, or running them from that location? Mirroring cuts spindle count in half which will severely limit the number of VM's you can comfortably run simultaneous.
THis is not for the VMs and all that.  This is just for the file storage.  I need to use more than 3 disks, I'm not doing RAID.  I can't tell if you guys are using the term RAID and drive pooling interchangeably.  I have no desire for RAID of any kind, but I'm interested in drive pooling for the purposes of having storage capacities larger than the biggest disks available (4TB).  SO I want 8TB drive pools or larger.