topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Thursday March 28, 2024, 6:44 pm
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: Drive Extender replacement due out in 2012. It's called Storage Spaces.  (Read 21921 times)

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
Every so often, one of the big players does something right.

In this case, the player is none other than Microsoft who has finally let us in on what their plans are for replacing the  Drive-Extender technology formerly found in Windows Home Server. From a recent article over at ArsTechnica, the answer is something called "Windows 8 Storage Spaces." A bit of technology that goes one better than what it is replacing. (full article here)

When Microsoft killed Windows Home Server's "Drive Extender" technology, we mourned its loss but held up hope that the company would persevere with the concept. The company has done just that with a new Windows 8 feature called Storage Spaces, described in a lengthy post to its Building Windows 8 blog.

With Storage Spaces, physical disks are grouped together into pools, and pools are then carved up into spaces, which are formatted with a regular filesystem and are used day-to-day just like regular disks.

Unlike RAID systems of old, but in common with other modern storage technologies such as Solaris' ZFS and Linux's btrfs, pools can use disks of different interface technologies—USB, SATA, Serial Attached SCSI—and different, mismatched sizes. New disks can be added to a pool at any time. Pools can also include one or more hot spares: drives allocated to a pool but kept in standby until another disk in the pool fails, at which point they spring into life.

Storage in a pool is then distributed among one or more spaces. Each space can have its own redundancy policy, with three kinds of fault tolerance offered: 2-way mirroring, 3-way mirroring, and RAID 5-like parity. With the mirrored options, a space's data is stored either twice or three times within a pool. With the parity option, the system will compute additional information and store this within the pool. If any disk in the pool fails, the data can be reconstructed using this additional information.

Spaces can be thinly provisioned, allowing the creation of spaces that are larger than the underlying pool. This allows potentially simpler management—a large "media" space for TV shows and movies could be created with some large size, say 50 TB, with only 2 TB of physical capacity in the pool. As more shows are recorded or downloaded, and space becomes tighter, additional drives can be added to the pool; the space will then use this extra capacity with no further configuration required.

The only drawback noted was the current planned implementation does not allow booting from a Storage Space disk group.

Perhaps the only fly in the ointment for most home users is that in Windows 8, Storage Spaces will not be bootable. The company says that guidance will be offered on how to partition disks so that a partitioned boot disk can be added to a pool, but that straightforward booting unfortunately won't be possible. On some levels, this is unsurprising: many advanced filesystem and storage systems are not bootable in their initial version, and Storage Spaces certainly won't be the first. On the other, it would certainly be a desirable addition, as it would ensure that even if a boot disk failed, your PC would remain operational.

Probably not that big a concern for most desktops. Especially since there is an announced partitioning workaround that will allow you to reserve a portion of a drive used in a Storage Space as a regular boot device. But I'm sure that little bump will be smoothed over eventually.

On the enterprise side, it's much bigger news - because yes!!! - Storage Spaces is also being targeted at enterprise servers. If this comes to pass, it will be a big day for Microsoft sysadmins everywhere. Because Storage Spaces has the potential to become that long wished for thing that will finally liberate server admins from that migraine headache called: RAID. It's a big enough deal that it would be worth considering a server upgrade for that alone IMO.

If Storage Spaces does make it to 'gold master', things are definitely looking up for 2012 AFAIC. :Thmbsup: ;D

JavaJones

  • Review 2.0 Designer
  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 2,739
    • View Profile
    • Donate to Member
Wooo! Can't wait to put Win8 on my HTPC. Not only will Metro actually make sense there, but Storage Spaces will rock for big (HD) movie storage!

- Oshyan

db90h

  • Coding Snacks Author
  • Charter Member
  • Joined in 2005
  • ***
  • default avatar
  • Posts: 481
  • Software Engineer
    • View Profile
    • Bitsum - Take control of your PC
    • Read more about this member.
    • Donate to Member
It has never went away, and exists in Windows 7 and Windows 2008/R2/Home Server 2011. They just removed the easy interface for it. Simply set up a 'spanned' volume and you can dynamically add disks as you want - including external disks. The intrinsic support for this never really went anywhere.

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
It has never went away, and exists in Windows 7 and Windows 2008/R2/Home Server 2011. They just removed the easy interface for it. Simply set up a 'spanned' volume and you can dynamically add disks as you want - including external disks. The intrinsic support for this never really went anywhere.

Interesting...does it also include the data protection features mentioned earlier? Disk pooling and spanning itself is no big deal. Safe data redundant spanning is a very different story. And redundancy/parity safeguards were what got left out when they dropped the old Drive Extender.

Correct me if I'm wrong, but from what I understood - this is the part that's very new to Storage Spaces:

Each space can have its own redundancy policy, with three kinds of fault tolerance offered: 2-way mirroring, 3-way mirroring, and RAID 5-like parity. With the mirrored options, a space's data is stored either twice or three times within a pool. With the parity option, the system will compute additional information and store this within the pool. If any disk in the pool fails, the data can be reconstructed using this additional information.

You can do some of that with software RAID in Windows. But that's for a declared drive array rather than a flexible pool.


 :)

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,646
    • View Profile
    • Donate to Member
You can do some of that with software RAID in Windows. But that's for a declared drive array rather than a flexible pool.

Yeah, the basic spanning, striping, & soft RAID stuff came out with the original Dynamic Disk on Win2k. The extended redundancy stuff only showed up on and currently exists for WHS. Sure partitioning is more On-the-Fly flexible on 7/08 but redundancy? Not so much.

Oh and about the RAID 5-like parity quote ... You're just twisting the knife on me aren't you.  :D

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
Oh and about the RAID 5-like parity quote ... You're just twisting the knife on me aren't you.  :D

Actually... no.  ;D

Like you. I work with RAID-5. A lot. (Used to almost automatically spec it for server clients too.) So if you're anything like me, you get enough pain from occasional RAID issues that nobody needs to inflict any additional agony on you when it comes to that.

 :Thmbsup: ;)

MrCrispy

  • Participant
  • Joined in 2006
  • *
  • Posts: 332
    • View Profile
    • Donate to Member
I believe this is very different from the spanning/dynamic volume support already present in Windows.

- data is written in 256MB slabs, which also include metadata. Thus the physical disks are readable in any other Win 8 pc, as long as you have enough to form a 'quorum'
- support for parity, mirroring and resiliency
- probably built on top of a new filesystem layer called ReFS, details on which are scarce

This is the work the WHS team was doing for DE v2 when they decided to scrap DE completely from WHS 2011. I hope its will include features such as Single Instance Storage and automatic error correction and detection (the 2 missing biggies) soon.

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
I believe this is very different from the spanning/dynamic volume support already present in Windows.

It is.  :)

I hope its will include features such as Single Instance Storage and automatic error correction and detection (the 2 missing biggies) soon.

The data relocation and recovery via parity and mirroring will be fully automated. Is that what you meant by automatic error correction?

Regarding single instances, I don't think that's in the cards. But it doesn't surprise me since that's more (in my experience) a feature of a data store (i.e. like Exchange's PST/OST + underlying database) rather than file store. AFAIK "single instance" presupposes the existence of a full database of some sort. It's a higher level of abstraction than a general purpose file management system.

IIRC Microsoft dropped across the board SIS in Exchange 2007. E2k7 only applies SIS to message attachments. And starting with Exchange 2010, it's gone completely since it negatively impacts I/O performance and interferes with the deployment of other planned features. Supposedly, SIS provided storage space savings of less than 20% of what it would be without SIS. So with the advent of inexpensive and huge hard drives, Microsoft has concluded that real world disk space savings no longer justify the performance hit and complexity costs of continuing to have SIS in Exchange.

But now I'm curious. Is there a general purpose file system/manager that enforces single instance data storage? I'm not aware of any. But that's not to say there isn't. (And I'm always interested in learning something new.  :))

DigitalPackratPaysOff.jpg


--------
XKCD's Randall Munroe Rules!!!!  ;D :Thmbsup:
« Last Edit: January 11, 2012, 12:26 AM by 40hz »

MrCrispy

  • Participant
  • Joined in 2006
  • *
  • Posts: 332
    • View Profile
    • Donate to Member
Windows server products such as Storage Server have SIS on all data. WHS also has it for backups.

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,646
    • View Profile
    • Donate to Member
The WHS backups did use deduplication to prevent multiple copies of (for example) the OS files from taking up excessive space. But I don't think that's quite what they're referring to. I haven't had a chance to play with WSS so I'm not sure what it does.

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
Not too up on WSS either since that's mainly an OEM technology for appliances and turnkey servers.

I can see deduplication for a backup, a message store, or anything similar since you'll be creating a database of files and calculating hashes anyway. The Cumulus media and digitsl asset storage servers used to (still do?) work like that. (Been a long time since I've seen one of those.)

I also noticed in the blog entry there was no mention of that. Not surprised. It would kinda eliminate some of the market for WSS Essentials if it included deduplication features.
 :)
« Last Edit: January 11, 2012, 08:48 AM by 40hz »

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,646
    • View Profile
    • Donate to Member
Not too up on WSS either since that's mainly an OEM technology for appliances and turnkey servers.

Bummer, I was hoping you were ahead of me on that one. I pulled the ISO's off the MSDN awhile back, but haven't had time to try to virtualize one to play with. I'd like to try playing with Windows Thin PC too (also on the MSDN).

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
Bummer, I was hoping you were ahead of me on that one.

Nope. Got my hands full enough I haven't gotten a chance to play with it. But since it's primarily Microsoft's take on an AD aware and enabled version of a NAS it shouldn't be that earthshaking a product - although it probably has some nice bells & whistles for domain admins.

Kier Thomas over at PCWorld dissed it a while back in a short op-ed piece:

Microsoft faces an uphill struggle. Why would a vendor implement WSS2008R2 when it brings with it a 25-user limitation and licensing fees that will push up the price of their hardware? This is especially relevant for smaller businesses, at which WSS2008R2-based products are to be aimed, as they're likely to have meager IT budgets compared to larger organizations.

From an original equipment manufacturer point of view, Linux might have higher initial setup costs--those pretty GUIs need a fleet of coders to create them--but this needs to be done only once for an entire product range. Additionally, some companies have struggled with the nature of Linux licensing, forgetting that they can't simply take Linux and add-in their own cool bits, keeping everything secret. Any changes to Linux that are then redistributed much be shared as source code.

The only reason I can see for WSS2008R2 ending-up in NAS devices might be to appeal to businesses so tightly in the grip of Microsoft that they can't possibly consider products by anybody else. The Microsoft brand still has some cachet in such circles, and using WSS2008R2 in a product will no doubt allow vendors to add "Designed for Windows 7" stickers to the box.

I don't always agree with Kier. But this time I think he pretty much nails it. WSS is something Microsoft brought to the table just to say "See! We have that too!"

... and to possibly make OEMs get around that common problem companies have with GPLed software (which I bolded in the above quote) when they want to benefit from FOSS - but not be required to share their code back as part of the deal.

..and to possibly make life a little easier for the CIO of a small firm that is required to make compliance assurances in a regulated industry like law or mortgage lending. I doubt a State examiner would have much to say about your software codebase if he/she saw Microsoft's name all over everything. Not that Microsoft's stuff is any more secure than anybody else's code if the number of patches and updates on Microsoft's website are anything to go by. But even if there were concerns, it's easy to point back to exactly who authored it. And that counts for 'major props' with regulators.

I personally think the main point of WSS was to provide something a little less flexible and more "professional sounding" than Windows Home Server for small business clients and SMB consultants. Or possibly provide an "out" for those who bought Small Business Server, ran out of storage space - and are now trying to figure out how to increase it without running the risk of messing with SBS and destroying their entire world.

ButNoooo.png

Plus, it's always expedient to try to take a little something away from Linux/BSD providers. Especially before people start thinking and asking pointed questions about stuff like...oh...y'know... licenses and prices...

Cuz Microsoft knows once you let Tux or Beastie in the door - there's just no telling what might happen...

But I'm sure they can can guess. ;D

8154-thumbnail.jpg   8153-thumbnail.jpg   8149-thumbnail.jpg

 :Thmbsup:

P.S. re: ThinPC - Yes! Definitely something I'll have to make the time for soon. That is a piece of tech I'm going to need to know a whole lot more about than I do right now.

I don't need a 40 hour week. I need a 40 hour day. :-\
« Last Edit: January 11, 2012, 02:04 PM by 40hz »

mahesh2k

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 1,426
    • View Profile
    • Donate to Member
You know what you got one convert this year.  :Thmbsup: I am literally trying to switch 100 percent to linux this year. I don't want to waste time with windows anymore. I have old computers which can't run modern windows but they can run slitaz or archlinux without much issues, virtualbox can solve some win dependent apps requirement. I guess anything nooh from apple and windows is starting to look useless to me. I played with w8 demo, apple and it's just not impressing me against linux.

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
You know what you got one convert this year.  :Thmbsup: I am literally trying to switch 100 percent to linux this year. I don't want to waste time with windows anymore. I have old computers which can't run modern windows but they can run slitaz or archlinux without much issues, virtualbox can solve some win dependent apps requirement. I guess anything nooh from apple and windows is starting to look useless to me. I played with w8 demo, apple and it's just not impressing me against linux.

I'll agree with you on that. If I were doing purely personal computing, or running a non-computer business, I'd be 100% on NIX by now.

A few things currently keep me from doing so:

1. My clients use Windows. So, in the spirit of "drinking what you serve," I also have to be a Windows user, both to stay on top of it, and have direct hands-on. Otherwise I wouldn't bother. ALL my personal stuff gets done on Linux. And all my personal servers are either BSD or Linux.

2. I don't pay the full tariff. I'm in the MS partner program. So I get access to something called an Action Pack. Which is a super inexpensive way to get access to most of what Redmond offers at an incredibly good annual license fee. If I had to actually buy this stuff at market price I'd be out of business since it would be too expensive for me to stay up on these products. Especially since (to repeat myself because it's important) hands-on counts for everything when you do field support.

Note: If you really are "in the business" of supporting Windows (i.e. you're a registered business with a taxpayer ID number) you'd do well to qualify for the program. It doesn't take much other than passing an easy test and agreeing to some very reasonable terms to get in. If you're in the business - get in on this. It's one of the few true bargains out there.

3. I sometimes need to be compliant with someone's proprietary model for a given sort of project.

If you have a music project where every other musician is using Sonar, you'd best be using Sonar too if you're floating sequenced files back and forth. You could do export/imports. And they might even work and be glitch-free. (Don't hold your breath however.) But time is money. And technical headaches get in the way of creativity. So insisting on an arguably (or more like 'possibly') better FOSS solution is still counter-productive. And it just gets everybody pissed at you anyway. Same goes for movie editors, or scriptwriting software, or spreadsheets. If you're working with accountants, the government, or big business - you use Excel. Each venue has it's preferred software tools. You'll find industry inertia or momentum drive tool choices more often than not. Very often it's not a good idea to try and buck the flow. Sometimes it's not even possible.

There are even conventions (mostly inertia-based) in local markets that need to be observed for pragmatic reasons. Example: network diagrams in consulting proposals get done in Visio around where I work. Always Visio. Just Visio. Nothing but Visio. Ever. To do otherwise is to brand yourself as an amateur with the clueless. But those same clueless people are also the terrific people who write out your checks. So you use Visio. Period. (And when all's said and done, Visio is a very nice diagramming tool. That much I will give them. Even if the current publisher wasn't the company who originally wrote it.)


*

So in this less than perfect world, I still really can't completely walk away from Windows or Apple unless I find something else to do full-time -and for money.

Not yet anyway. :(

walk_away1.jpg

But I'm working on it... 8) ;)


superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
This is the thread I've been looking for.  Thanks 40.

Storage Spaces is the technology that I've been dying for.  I'm currently studying the details to learn what is going on.  I have some simple questions:
1) If I pull out one of the disks and plug it into a different computer, is that readable?  That's one reason why I never liked RAID.  I like the idea of pulling out a disk and being able to still read the files/folders there.  if this is not a big deal, I'd appreciate an explanation why.

2) If I already have a disk with stuff on it, can I stick it into the Storage Spaces array and use it without needing to format it or otherwise move/lose the content already there?

3) What happens when a disk goes down?  Does it get rebuilt RAID-style?  Am I restricted on the new type of disk I can replace it with?  If I have another Storage SPace array where the same data is backed up to, do I just recopy the files until they are both synchronized?

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Got some answers to my own questions:
Architecture

A Storage Space operates at the block level while it should work at the file level instead. The benefit of potentially high read speeds (chunks from larger files are stored on different drives and can be read concurrently) is not relevant in SOHO scenarios. But the increased probability of error is.

By splitting files into chunks (called “slabs”) and striping these across disks, it becomes:

    impossible to access a single disk’s data without all other disks being accessible
    (you cannot take a disk from the pool and read its data in another computer)
    impossible to add existing disks to the pool without losing the data stored on them
    (when adding a disk to the pool, it is initialized and formatted – importing data into the pool by adding a new disk does not work)
    impossible to convert between different redundancy types (none, parity, mirror)
    much more likely that the entire system breaks because failure of a single drive destroys all data on all disks (two drives with parity or one-way mirror)
    (in file-based solutions, in case of disk failure only the data on the failed disk is lost, the remainder of the pool remains unaffected)
http://helgeklein.co...gs-and-design-flaws/

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
Another option for Storage Spaces, and the reason why i mention it is to address my issue of "pulling out the drive and using it on another machine" is:
http://www.greyhole.net/

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
After doing some more research, it seems like the real answer to simply pooling a bunch of drives together is Drive Bender.  This is my favorite feature:
Non-destructive

One of the key Drive Bender features is its non-destructive file system technology. In short, this means that all drives within a Drive Bender pool are utilizing standard NTFS format and file structures. This is to such an extent that a drive can be pulled from the pool and read on any machine capable of reading an NTFS formatted drive. More over, drives attached to the pool can be done so without modification, if a drive is added (not merged) that contains existing data, that data will remain untouched.

And it runs on any version of Windows, so no need to wait for Windows Server 2012, etc.  Unless something else comes up, this looks like the winner.

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
@SB - I don't know of any general purpose pooled storage system that recommends you add and remove drives as a matter of course. That's more what external USB drives are intended to be used for.

Because these systems need to compute parity data or take snapshots to work their magic, arbitrarily pulling drives out isn't a good idea - even if you can get away with it. Nor is it going to work they way you'd want it to since the minute a drive disappears from the pool, the system will begin reconstructing the missing disk's files on the drives that are still installed. The other thing is the pool manager uses all the resources in the pool. So there's usually no telling on which disk a specific file or directory is located. It could be spread over several physical drives.
 8)

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
@SB - I don't know of any general purpose pooled storage system that recommends you add and remove drives as a matter of course. That's more what external USB drives are intended to be used for.

Because these systems need to compute parity data or take snapshots to work their magic, arbitrarily pulling drives out isn't a good idea - even if you can get away with it. Nor is it going to work they way you'd want it to since the minute a drive disappears from the pool, the system will begin reconstructing the missing disk's files on the drives that are still installed. The other thing is the pool manager uses all the resources in the pool. So there's usually no telling on which disk a specific file or directory is located. It could be spread over several physical drives.
 8)
What do you recommend as the best way for me to create an enormous 10+ drive pooled system?  I'm trying to avoid RAID because I don't really understand it, and I have disks of all different sizes and models.

How about this?  Do you see anything wrong with this setup:
--don't pool, don't raid, all drives are just regular drives
--let's say i have a "video" folder on two drives and i want them to appear as one, I'll just use the Windows Libraries feature.

That's my ideal setup because then I have regular drives without the headaches of raid or pooling complications.  But I also get to group folders from different drives together.  But if I'm overlooking anything, please let me know.  I'm in the middle of creating my server architecture diagram, so I'm doing this as we speak!

superboyac

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,347
    • View Profile
    • Donate to Member
As I'm reading more, another option to consider is Distributed File System (DFS):
http://en.wikipedia....stem_%28Microsoft%29

So I'm currently debating:
DFS
Drive Bender
Standalone normal drives
RAID

I'm leaning towards DFS, but I now have to learn how it works.

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,646
    • View Profile
    • Donate to Member
I'm leaning towards DFS, but I now have to learn how it works.

Not the way you want it to. The DFS root contains links to discontiguous shares in a single namespace. You don't want to be pulling and swapping drives with it, because as the drive letters change your data will be playing peek-A-boo (and with very little peek...).

I love DFS for a server for its flexibility in letting me modify storage access within a single drive mapping. But the targets really need to be static. Effecting redundancy via distributed DFS will take way more hardware that you want to/need to/should buy.

40hz

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 11,857
    • View Profile
    • Donate to Member
I've heard good things said about SnapRAID but I can't personally vouch for it since I've never installed it. (Warning: it's CL so it's a little geek.)

This page on their website also lists several alternatives to Snap RAID that might be worth investigating.

Q: Is data protection or media pooling the more important feature for you?

« Last Edit: July 17, 2012, 07:25 PM by 40hz »

4wd

  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 5,641
    • View Profile
    • Donate to Member
I mentioned a couple of things here.

SnapRAID - Filesystem agnostic parity based data redundancy, (Linux/Windows).
Elucidate - A GUI for SnapRAID (Windows).
Liquesce - Drive pooling for Windows.