NTFS Compression hack under XP - is it safe and is it worth implementing?

[Darwin]

The subject line says it all. In another thread this hack is exposed as being the "secret" being sold by a software distributor for $7.50 as a downloadable, password protected pdf. I'm wondering what the pros and cons of implementing such a hack are...? I'll be researching this myself and will post back here, but don't know how soon I'll be able to do so as I've a full teaching schedule to prepare for!

[Josh]

NTFS compression, as I said in the other thread, is best used at the folder level. DONT USE IT ON THE SYSTEM DRIVE (I dont think this can be done anyways, but if it can, dont). If you impliment this on the drive level (I.E., choosing the "Enable compression" option at format time), make sure its on a drive where lots of uncompressed material will be stored as doing so on a drive with MP3's, PDFs and other already compressed material WILL slow down performance especially if the files are larger than a few megabytes.

Folder level is the best method for use of compression since you can select which folders are compressed and which arent.

[f0dder]

First: the other topic, discussing the scamsters at DiskTrix claiming that NTFS compression is oh-so-über-fantastic (and not even revealing that their "OMFG SPEED UP WINDOZE!!!11! one one" trick is NTFS compression until you pay them ca$h).

With that said, NTFS compression isn't a hack, and it can have some good uses; but just like Josh, I'd never enable it systemwide. For some things it's decent enough though, like compressing the contents of the PlatformSDK include folder. I made a copy of this folder, enabled NTFS compression for it, made sure that both the compressed and uncompressed files were defragmented (sysinternals' contig tool), and then rebooted so I was sure the filesystem cache was empty.

Here's the results of "md5sum -b *" in the folders, timed with "timeit.exe" from the Windows Resource Kit:

Version Number:   Windows NT 5.1 (Build 2600)
Exit Time:        3:19 pm, Tuesday, September 4 2007
Elapsed Time:     0:00:02.703
Process Time:     0:00:00.484
System Calls:     136548
Context Switches: 130067
Page Faults:      13608
Bytes Read:       102010609
Bytes Written:    0
Bytes Other:      128204

-compressed

Version Number:   Windows NT 5.1 (Build 2600)
Exit Time:        3:20 pm, Tuesday, September 4 2007
Elapsed Time:     0:00:04.437
Process Time:     0:00:00.375
System Calls:     135903
Context Switches: 132015
Page Faults:      13701
Bytes Read:       102214521
Bytes Written:    0
Bytes Other:      133980

-uncompressed

So about twice the speed to read in the files... 10,000rpm raptor drive and AMD64x2 4400+ CPU. Disk usage: 49.299.474 bytes, taking up 18.926.984 bytes on disk, volume cluster size: 4kb (gains/savings likely to be different with different cluster size because of how NTFS compression works).

Thing is, the PlatformSDK include folder is "pretty huge" considering it's plain text, but even that kind of "plain huge" is <50 megabytes, and under normal use (ie., compiling apps) it's probably going to be cached most of the time, so it's not that big a deal imho. And you'd be hard pressed to find real-life situations where you can get better results than this and actually feel the effect.

Gonna reboot and post stats for the PlatformSDK "lib" folder too...

[f0dder]

Okay, results for the Lib folder: 23.654.752 bytes, taking up 9.768.564 bytes on disk with compression.

Version Number:   Windows NT 5.1 (Build 2600)
Exit Time:        3:32 pm, Tuesday, September 4 2007
Elapsed Time:     0:00:00.750
Process Time:     0:00:00.156
System Calls:     29265
Context Switches: 26811
Page Faults:      3980
Bytes Read:       29256002
Bytes Written:    0
Bytes Other:      27048

-compressed

Version Number:   Windows NT 5.1 (Build 2600)
Exit Time:        3:32 pm, Tuesday, September 4 2007
Elapsed Time:     0:00:01.359
Process Time:     0:00:00.109
System Calls:     29211
Context Switches: 27563
Page Faults:      3973
Bytes Read:       29240194
Bytes Written:    15928
Bytes Other:      30128

-uncompressed

[Carol Haynes]

That last one is an interesting comparison - more bytes are read from the compressed version that the uncompressed version? How does that happen? Also CPU time is 50% more in compressed mode - even though the overall time is less - this would suggest that if you have other CPU intensive activity in progress then there is likely to be a system hit which affects ither activities as well as the overall access time.

[f0dder]

Well, a word of caution: these datasets were very small, so do keep in mind that there's going to be some fluctuation. You'd need to test on larger datasets before measuring CPU time taken - also note that I didn't do any affinity limiting either, further introducing risk of inaccuracy.

Obviously decompression isn't going to be free, but I doubt it's going to be more than a few percent actually (compression is certainly going to be more expensive than decompression though).

Dunno about the bytes read - that _is_ interesting... ought to do some more intensive testing and more than just a single test run for each case. I wonder if there's some way to ask Windows to discard it's filesystem cache, instead of resorting to the "allocate all physical memory" trick. Rebooting the system between each test run _IS_ too tedious

[Carol Haynes]

There must be a way to flush the cache ... coding snack?

This thread may be of interest:

http://forum.sysinte...sp?TID=5677&PN=1

EDIT ...Does this utility do the job?

http://www.microsoft...lities/CacheSet.mspx

Clearing the Cache's Working Set
You can force the Cache to release all of it's pages by pressing the Clear button. Note that the Cache can grow again as necessary, and that this is not the same as flushing the Cache - pages that were assigned to it are simply made available to other programs and can be reclaimed by the Cache.

[f0dder]

Will have a look at that carol, thanks.

Flushing write cache is easy, but I didn't/don't know of a way to empty the read cache...

[Carol Haynes]

That utility seems to flush the system cache - not the read cache.

I haven't tried it - what are the consequences of flushing the system cache while windows is running - presumably it should all take care of itself with a certain amount of disk thrashing as Windows grabs back essential files to the cache from disk?

[f0dder]

"clear" from cacheset didn't seem to do much about read cache. Memory usage goes down, but then starts going up again - timings from md5sum'ing a ~500meg file clearly shows that the file is still cached, though.

Dunno if the sysinternals forum thread helps anything either, even if I did a FlushFileBuffers() on all file handles in the system, I'm pretty sure that only takes care of flushing the write buffers (at least according to MSDN/PlatformSDK).