Implementing Leopard features for Vista?

[MrCrispy]

I'm quite impressed with Leopard's usability features such as QuickLook and TimeMachine. Yet, I am also dissapointed that all of these are possible today in Windows (and specifically Vista), yet Microsoft and 3rd party vendors seem to lack the ability/desire to use them.

TimeMachine
.. is one of Leopard's most hyped features, and with good reason. It finally offers normal users a simple, consistent and powerful way to do backup and system recovery. But if you look under the hood, Vista has had this technology since long, ever since the Volume Shadow Copy Service (VSS) debuted in Windows 2003 server.

VSS is technically better and much more powerful than TimeMachine - it can offload the copies to hardware if present, it works on a block rather than file level, its districubted and extensible. But in typical Microsoft fashion, the UI (previous versions tab and System restore) is not nearly as easy to use, and much of the features are hidden since apps don't make use of them. e.g. in Leopard AddreessBook is aware of system snapshots and can search them. In Windows the only apps that seem to use VSS are imaging and backup programs.

I don't care for the starry backgrounds, but why can't this be implemented on Windows fairly quickly?


QuickLook
More than anything else, this is the one thing that stands out most in daily use. Yet most reviews never mention that Vista already has this too! Its called the preview pane and any app can register handlers for it, much like in OSX. So why don't we use it more - because the UI is clunky, its only in explorer and and once again, other apps don't take advantage of it.

Stacks, Spotlight etc
To me, these are not sufficiently different from the equivalent in Windows or that useful, so I won't discuss them.

[nontroppo]

Time machine was built to solve a very real problem:

Eighty percent of Mac users said they knew they should backup their data. (This is scary already. Only 80 percent?) Twenty-six percent said they do backup their data. That actually doesn't sound too bad until you get to the next question. Only four percent backup regularly.

source
This is not some weird geek issue for sysadmins, but a profound fracture in how people use computers. With that user focus (4% regular backups!), Engineers worked to do something about it. Windows users probably have similar backup rates, and yet the technology developed was never exposed to the majority of its users. Power users often ignore the UI as irrelevant (bah its just eye-candy) but the way features are presented to users is critical, as VSS so clearly demonstrates. Apple engineers worked both on the underlying details, but also a UI that works. I don't think VSS had any such user focus, it was probably built as an IS feature. No one at Redmond had the bright spark to allow normal users a clean way to use it too.

So why don't we use it more - because the UI is clunky, its only in explorer and and once again, other apps don't take advantage of it.

OS X is cleanly built around the idea of unified interoperable thin frameworks. Services are offered by the OS and used by applications. Applications are encouraged to allow interoperability by offering exposed dictionaries of things they can do to any application that asks. Developers are encouraged to use such unified services. Tom Yager expresses it this way:

Windows and Linux are designed from the core out, which is to say that they are all about layered kernels, system calls, and APIs, with each layer's purpose being to abstract the layers below it. The layers grow thicker; when a layer gets unmanageably thick, a pretty new abstraction layer is created so that people don't have to deal with the ugly one. ... Everyone plugs into OS X through the frameworks, and below that lies a stable, thin, simple, and well-documented system stack. It is not the frameworks' job to abstract lower levels of software. From a developer's point of view, the frameworks are OS X. When developers write to Apple's frameworks, they inherit cross-application integration and operational and interface consistency with no effort.

source

Even before Leopard came out, the main apps I use had added support for Quicklook! I think quicklook is more clearly thought out in UI terms than the various preview mechanisms available in several versions of windows back to 95. The proof is in the pudding; and that is that Quicklook will be a core UI feature for most users whereas Preview has been available but woefully underused on windows.

Spotlight *is* a good example of this. I used X1 / GDS on my Windows machines before my forced move to OS X. So I was fine with a windows equivalent and it was a useful tool. Yet because Spotlight is a core OS service, and cleanly exposed in a framework, many of my applications use it. That makes spotlight far more useful for me as a user. Developers can all build on it and it just becomes more valuable to users as a result. Vista now has a search mechanism (though a much less robust metadata framework), but what is a developer to do when there are so many other possible search services to support (and most users are still on XP)?

In conclusion, the two examples you give are fairly clear cases where an attention to interoperability, user focus, and cleanly exposed frameworks benefits both developers and users.

---------------------------------------------------------
EDIT:
As I've not installed Leopard yet I've not been able to compare its functionality, but just reading through the Ars Technica review, Time Machine is actually more advanced than VSS as far as I understand. One can recover individual items within a file store, for example a deleted entry in an address book, or individual mail. VSS would force the user first to know what the file name that stored the info was, but then restore would lose all changes made subsequently. It would require manual hacking to get the data out. Selective item revisions without data loss is a critical feature of a backup system, and thus technically VSS could not provide such complete functionality even if the UI was reworked IIUC.

[MrCrispy]

nontroppo, I believe you misunderstood my post. My entire point was that even though Windows has very similar (and in some cases maybe better) technology under the hood, the implementation and the UI leaves a lot to be desired and is not as useful and easy to use as Leopard. And I was hoping that there are some apps that fix this.

The consistent design of apps and the system itself in the OSX ecosystem is one reason why everything works so well together. In Windows there are a million different API's and frameworks, and even though Microsoft is very clear (most of the time) about what they want you to use, developers (both inside and outside MS) don't always follow it.

[nontroppo]

I may not have been clear enough  ; VSS cannot implement a system like Time Machine technically — rolling back to specific items not just data stores. VSS neither integrates into search indexes (ala Spotlight), and so cannot support the ability to search through past items *within* individual applications (i.e. you find old mails within Mail.app, not needing to open Time Machine etc.). Time machine is not just a pretty RCS, it locks into the file system and core OS services like Spotlight. VSS could probably be glitzed up to something more useful than it is now, but it may need core plumbing changes to do what Time Machine does. I suspect there are some versioning file systems additions to NTFS that can do that for windows, but again this is bolt-on to the OS, thus won't be universally supported.

The point that some kind of preview system has been available since Win95 but largely unused suggests the windows ecosystem cannot yet sustain a useful quicklook-type feature. I think Vista is probably close though (though someone suggested it can't generate previews for Office Docs unless Office 2007 is installed?), and they just need to rethink the UI. They should ditch the preview pane and use a pop-up window, and push the API harder. One caveat though is how previews are stored and where, HFS+ has pretty solid metadata mechanism (and many files are actally bundles thus store Quicklook info there), how does Vista handle this?

[Carol Haynes]

Acronis TrueImage uses VSS and allows incremental images to be built and 'mounted' into the filesystem as virtual drives (you can assign the drive letter). Those drives can be read only or opened as writable dives. I see no reason why they should not be indexed too using your favourite desktop search engine and/or Locate32 so that multiple versions of files can be accessed.

Also if you want versioning of files that are indexable and searchable why not just use something like FileHamster to keep versioned copies. Your desktop search tool would then potentially include all versions and be continuously updated. You could then use FileHamster to revert to the desired copy as required.

OK it isn't the same as Leopard function but if I understand what you are both writing it would be functionally equivalent or at least a simple and servicable solution. The only thing that would need to be thought about is the amount of disk space required and potential system hit if you want to apply version control to more than just your data files. Is there any real point in going beyond data files anyway - just keep them separated from your system partition and use incremental or differential backups to maintain backward system mobility.

[MrCrispy]

Carol, what you propose is quite possible except for the extra layer of indirection needed to mount the image so it appears as a filesystem. That is what TimeMachine does so nicely, the backup snapshot appears to everyone else as a regular copy of the data so its easy to search, index and view. With a VSS snapshot, I must use some kind of Volume explorer thingy to do so and its not always available, so it won't get indexed automatically.

I haven't used incremental snapshots in Acronis but I wonder if mounting them gives me access to the complete filesystem, or only the files that changed in that snapshot?

W.r.t. VSS vs TimeMachine, using hardlinks is quite clever, but as mentioned in the review, TM does not work at the block level which is a big disadvantage. Microsoft has block level shadow copies and it makes a huge difference. Windows Desktop Search could also be rewritten to search any 'previous versions' in addition, that would require no core OS changes.

[Ralf Maximus]

I haven't used incremental snapshots in Acronis but I wonder if mounting them gives me access to the complete filesystem, or only the files that changed in that snapshot?

I use Acronis in incremental mode.  So long as all of your backup files are present, you have a complete image -- not just the files that changed.  You will get a complete snapshot of the whole filesystem as it looked when the incremental was taken.

And yes, it's powered by magic.

[Carol Haynes]

Yes if you mount numerous increments you get the complete drive for each increment under a different drive letter at the point that the increment was made.

If you use something like X1 or Exalead it should be possible to index your files in each increment just once (but make sure each increment is allocated a different drive letter) - then the increment doesn't need to be mounted to search the image - obviously it will need to be mounted if you actually want to view the files but that just involves a right click on the increment file and select mount from the menu.

[nontroppo]

So to get something close to Time machine, one needs to mount drives for each snapshot (TM does hourly (24), then daily (~29), then weekly (...) snapshots). I suspect one will run out of drive letters before long? I think one would need to break that useful timing scheme, so only allow weekly for two months then monthly (or can windows use "silent" drive letters)? This doesn't seem like a serviceable solution for general users unless some changes are made to the requirements of having to mount drives - is there any other way to do it?

Could one not use some program which catches requests to mount a drive and uses a hash of that mount volume to find the right mount - thus only one drive letter would be needed?

Is there any way to get Vista's desktop search to index those revisions (even as mounts) currently?

r.e. block mode: as I understand it, Apple had originally wanted to use ZFS to do the grunt work for block mode, but technical difficulties forced them to withdraw ZFS support for Leopard final (which is read-only for users and experimental read/write for developers). Indeed the weakness currently is for monolithic files, of which VMs are the clearest example. I wonder if they can't do some sort of delta mechanism on top of the hardlinks - deltas don't need block-mode IIUC.

[Ralf Maximus]

So to get something close to Time machine, one needs to mount drives for each snapshot (TM does hourly (24), then daily (~29), then weekly (...) snapshots). I suspect one will run out of drive letters before long? I think one would need to break that useful timing scheme, so only allow weekly for two months then monthly (or can windows use "silent" drive letters)? This doesn't seem like a serviceable solution for general users unless some changes are made to the requirements of having to mount drives - is there any other way to do it?

That's the theory.  In practice, I usually only keep a week's worth of incrementals around at a time.  I figure if my Windows gets trashed and I don't notice within a week, there's something wrong with me.  So to keep a week's worth of incremental backups online would require (for me) a maximum of seven drive letters.

But having all that available all the time seems wasteful.  That's a lot of horsepower being used for something that might be needed but probably never will.  If I do need to delve into the past, then I can mount an image -- that "on demand" capability is a saviour during times of crisis, but how often do things really go to hell?  In my experience, twice in the past two years.

And as frequently discussed on other threads, more transparent means of incrementally backing up work files exist: File Hamster, for instance.  Apply a solution like that to a few critical folders and 98% of anyone's needs should be met.

I perceive TimeMachine as a wonderful "shiney object" that will make casual users purr, but beyond that it's more of a marketing tool than anything.  If enough heat & noise is generated about it then Microsoft will respond, and Vista's copycat version will be technically better but less elegant to use, and in the end everyone will still rely on traditional backup strategies.

[Carol Haynes]

So to get something close to Time machine, one needs to mount drives for each snapshot (TM does hourly (24), then daily (~29), then weekly (...) snapshots). I suspect one will run out of drive letters before long?

-nontroppo (November 08, 2007, 05:46 AM)

I suspect one will run out of disc space on a Mac if it does hourly snapshots - Apple aren't exactly generous with their supplied hard discs anyway.

Seriously though there is no reason why you couldn't do hourly snapshots with Acronis TrueImage - and periodically mount and index the images.

I am not quiote sure what TM indexes consist of - are they really like a desktop search of document contents or is it just file names? If it is indexing 24 snapshots a day it must make a huge performance hit both in terms of CPU usage and disc space usage !!!

Maybe my other idea with FileHamster (or another file versioning utility like Adobes Version Cue) would be a better approach on windows. If you set it to watch whole drives then you can say keep only the n most recent revisions of files and then every file is instantly version controlled over the number of revisions you require. If you have a decent desktop search tool like Copernic then all of those revisions would also be continuously updated in the index and it would all be pretty transparent.

If you just want a system to be able to roll back just set up a scheduled incremental or differential imaging pattern but there is no real need to keep the contents indexed on system files etc.

[nontroppo]

I am not quiote sure what TM indexes consist of - are they really like a desktop search of document contents or is it just file names? If it is indexing 24 snapshots a day it must make a huge performance hit both in terms of CPU usage and disc space usage !!!

-Carol Haynes (November 08, 2007, 07:19 AM)

The Ars Technica and Prince McLean review delves into the details. But the basic point is that the filesystem transparently monitors file system events anyway. Any app can use these notifications (instead of forcibly watching for chages, just register to receive updates). This arrived with Spotlight, but has been made a public API in Leopard. This gives a low overhead mechanism[1] to know what to backup without specifically scanning the FS. It will ignore temp files and the like already, but if you want you can manually micromanage what is backup up IIUC. It uses something called multi-links (like a hard-link):

http://www.appleinsi...hine.html&page=3

to keep each versioned system complete but not use up more space than needed. Every day, it drops the previous day's hourly backups. Every week it drops the previous week's daily backups. Thus is balances the fact that you probably want to restore something you deleted quite recently with the possibility of still keeping older versions around.

Maybe my other idea with FileHamster (or another file versioning utility like Adobes Version Cue) would be a better approach on windows.

-Carol Haynes (November 08, 2007, 07:19 AM)

Well, I love Filehamster and depend on it in Windows. But it isn't a versioned recovery tool, it is a personal versioning system. I suspect part of the role of TM can be served by it, though I'm not quite sure how comfortably it would integrate with a desktop search.

Again I suggest that, for most users, the interface does matter. Hobbling together 3 or more utilities, each with a different UI, not servicing the purpose, is not the same experience as a tool built for the purpose. Unifying disaster recovery, versioned filing and desktop search throughout the OS is a usability win. That will reduce the discrepancy between those who know they should back up with those who do.


----
[1] I use a program that builds lists of events, FSEventer to see changes to the whole FS in real time, the kernel overhead of that app tapping into FSEvents is not measurable.

[Carol Haynes]

I used to use Powerquest's V2iProtector (which was their corporate imaging solution and image compatible with Drive Image). Unfortunately is was discontinued when Symantec bought out Powerquest. V2iP had a similar setup in that there was a low level service that monitored disk activity to speed up incremental image creation. It wasn't 100% there by the time the product was ditched but it was a similar idea to what you describe.

TM sounds interesting and a neat solution (assuming it works consistently and doesn't get screwed up when something else in the system goes wrong). I get the impression that a combination of incremental and differential backups using something like Acronis TrueImage (there are plenty of other apps that could probably do this too such as Genie Backup Manager if you want to monitor very specific or multiple file/folder sets) would effectively give the same rollback function. You can use the built in scheduler in TrueImage to specify hourly incremental backups and only retain 24 increments so that the oldest is dumped after day 1 on a rolling program. You can also specify a daily differential backup which would act as a daily rollup of all the increments - and again only retain 7 old differential images. Periodically you would need to start with a new base image - but I'd guess TM would need to occasionally move day 0 too otherwise the system would fill up within a few weeks.

[MrCrispy]

TimeMachine will not backup to the same disk, like VSS will do. Nor will it backup over the network, or to anything that is not a dedicated disk or is not HFS+. In that respect it is more limited but that really doesn't matter. Its so simple to setup and use that for many users, its the first viable backup scheme.

If I have a baseline snapshot in TrueImage, and then 10 incrementals, I would expect to mount one snapshot in a single drive letter and then be able to browse my entire drive. This is what VSS and TimeMachine both allow you to do. The difference, and its a key one, is that with Vista you have to know where to start - the exact folder or file, and then you can browse. With TimeMachine, its all interactive and you can choose to browse anywhere. Its possible because of the special hardlink support for directories that Apple implemented. This is also possible in Linux today with a number of programs such as rsync, rsnapshot and ext3cow. But of course none of them are as usable as the Windows/OSX versions.

To me TimeMachine is a moderatley impressive backend (hardlink for directories, fsevents) coupled with a fantastic frontend (the flashy UI, integration with AddressBook and Spotlight). I still believe that technically the backend in Windows is superior and I see no reason not to have that frontend too.

I don't think backup programs should use propietary file formats. Keep them as files, compress them transparently (if on Windows) and that way other apps and OS services can use them. You know what, I feel sufficiently strongly about this that I may start writing my own

[nontroppo]

MrCrispy: 

As a challenge, I will state (tongue-in-cheek but in the spirit of my above posts) that you will be unable to easily write a TM clone using the Win32 API. And I'd genuinely love to be proved wrong 

[nontroppo]

Actually, some Linux users are working hard to prove me wrong:

http://code.google.com/p/flyback/

Currently it uses rsync, thus it has to scan the whole drive on each backup. The latest Kernel contains inotify, which affords the same benefits as FSevents. They also cannot use multi-links as Linux doesn't cleanly handle hard-links to directories.

My critical caveat though, which is proper system integration, is also unknown, but it should be possible to hook this into Desktop search if they both use inotify.

Note, using rdiff-backup rigged up in this way, will also get you a block-level mechanism.

See also:

http://blog.interlin...nc_time_machine.html
https://wiki.ubuntu.com/TimeVault

Which I assume is also applicable to cygwin...

[f0dder]

There's a thing I've been pondering about wrt. timemachine/whatever use of hardlinks to avoid making backup of unchanged data...

What happens when the file *is* modified, and you look at the "backed up" hardlink from yesterday? Won't you get the changed data, then?

[MrCrispy]

The hardlinks point to the data in the backup drive. If the data is modified, the new file is going to get copied as well when the backup is made.

[f0dder]

The hardlinks point to the data in the backup drive. If the data is modified, the new file is going to get copied as well when the backup is made.

-MrCrispy (November 16, 2007, 11:42 AM)

Ah, righty - missed that part. Hardlinks obviously don't work across partition boundaries either