Other Software > Announce Your Software/Service/Product

The MagicRAR Drive Press Challenge

<< < (2/4) > >>

f0dder:
Ah yes, almost forgot this:
6. "The software probably does nothing else than force the compression flag on all files, even on those, where it would make no sense"

False. If you indiscriminately compress all files on your computer, you will actually end up with an unbootable system. Drive Press is completely safe to use and will not compress files that will jeopardize your operating system's integrity or ability to boot.-simonking (January 07, 2013, 01:30 PM)
--- End quote ---

After running DrivePress, FindCompressed "Found 1008 uncompressed files in 131695 items examined." - a bit of this seemed relatively random (Vmware Tools, some WinSxS folders), Windows Defender was also untouched (I wonder if MS applies special protection to those folders?), and obviously some "heavily locked in-use files" (like registry hives) were untouched.

Also, as I previously mentioned, there's indeed some files you shouldn't compress if you want a bootable system. Those are a very small subset of the files left uncompressed by DrivePress. Searching for "ntldr" in the executable gives the following list, which makes sense:
{ "NTDETECT.COM", "ntldr", "boot.ini", "bootmgr", "BOOTNXT", "\Boot\" }

simonking:
1) If you run MagicRAR Drive Press on Windows XP, or for example Server 2003, you will see that there is no difference in free space gains between using MagicRAR Drive Press and Windows itself to compress the drive. This is because this bug does not exist on NT 5.x OS's. It was introduced for the first time in Windows Vista, and has remained on all Windows versions (including Server editions) since.

As you have correctly discovered, a big part of this difference (though not all) is explained by the Windows shell failing to temporarily acquire and then restore permissions to protected file system areas. This is a bona-fide bug, because as you have seen for yourself, compressing these additional files is completely safe, and was already a matter of fact on older Windows versions. There is nothing preventing Windows Explorer from doing exactly what MagicRAR Drive Press is doing for you here; in fact that is exactly what it should have been doing in the first place.

2&3) You would need an SSD to see this for yourself, but no - these claims of yours are still false. Because most modern SSDs have an incredibly high number of IOPS, and because NTFS compression has very low CPU overhead as you have correctly determined, it is possible to dramatically accelerate the conversion process of an entire hard disk by maxing out the worker threads available.

In my experience, one CPU core (even if it is only an HT core) can handle two NTFS compression worker threads. So this is why the MagicRAR Drive Press Options Window offers to create up to double the number of worker threads as you have CPU cores available. Whether or not you use all of these depends on how fast your hard drive is. I would not recommend more than two-three threads with a mechanical drive, based on how fast it is (say 1 thread for 5400 rpm or slower; 2 for 7200 rpm or slower, and 3 if you have a 10k rpm drive - maybe 4 if you have a 15k rpm drive).

With an SSD, you can go really crazy though! For example Intel 320 series 600 GB SATA II SSDs seem to be able to handle 12 threads just fine. Now, even with Windows having a lot less files to compress, you can imagine that with 12 times the firepower, MagicRAR Drive Press just comes out on top of the speed race: Assume that Windows has about 1/3rd the job to do (roughly accurate, per our claim that MagicRAR Drive Press triples Windows compression). If MagicRAR Drive Press has to do 3 times more work, but can do it 12 times faster; this still works out to about 4 times faster globally. So this is a happy case of you having your cake and eating it too: You save three times more space, and four times faster at that! Does that sound like well built software to you?

4) False again. NTFS compression does increase fragmentation on hard drives, and this has nothing to do with MagicRAR Drive Press, since it is an innate disadvantage (and quite probably the only one) of using NTFS compression. However, a fragmented hard drive has absolutely zero impact on an SSDs performance. Fragmentation slows down your read/write speeds, because the drive head is constantly waiting for the platter to spin to all kinds of different places when reading/writing data. It is exactly this overhead that SSDs do not have, and that is why they feel so fast.

Because an SSD can read from/write to all parts of the drive at the same time (think of a hard disk platter rotating at infinite speed), that is why fragmentation is of absolutely no consequence for SSDs - be it NTFS compression induced, or the "normal" fragmentation that happens on NTFS inevitably. There is no delay, because all areas of disk are equally accessible at all times. In fact, this random access speed boost is the main reason why SSDs feel so fast. If you compare sequential access speeds between SSDs and mechanical drives, they're not too far apart. There's certainly no 10-20 times difference in those peak read speeds. It's only when you start comparing random access speeds that SSDs are many orders of magnitudes faster than mechanical drives. That is the main reason why its such a thrill to use them - you can keep throwing more and more work at the computer, and it just won't slow down.

A few closing points:

a. Do NOT attempt to manually acquire file permissions just to be able to compress them. Doing this will create a huge security hole on your system (one that MagicRAR Drive Press does not create, because it restores all permissions as has been confirmed in this third party report). And because virtually all areas of your system folders have unique permissions, it will be near impossible for you to manually restore permissions correctly. You may find yourself unable to install software on your computer, for example, if you mix up permissions during the restore phase.

Again, this is why properly speaking, this is considered to be a bug in Windows - one that is practically impossible to workaround manually without third party software that does the job automatically.

b. There will always be some files/folders that would be locked by the system/applications, and as such incompressible. If there is demand for it, we could also automate the conversion of those parts by building a boot time version of MagicRAR Drive Press - however, in my research, the additional space savings would be negligible.

So while the MagicRAR Drive Press Challenge technically remains unmet, I would be delighted to give f0dder a free key for the full MagicRAR 8.0 product, in appreciation of his time. It's the least I could do to thank for an open mind.

f0dder:
1) If you really belive it's a bug that the Shell doesn't temporarily remove protection of critical system files, you should file a but report on MS Connect, instead of making spurious claims in your marketing material - I'm pretty sure this is a by-design decision from Microsoft. I do agree it's probably harmless to compress those files, but calling a security feature a bug is misleading marketing, IMHO. And you deliberality keep your wording vague enough (combined with your "three times smaller", which is obviously only valid if there's not much else on the disk than Windows) to give the impression that the "bug" would be somewhere else (like, the core NTFS compression routines). THIS is why I'm pursuing this aggressively - you're using snake-oil salesman tactics. Which is a shame, since you obviously do get a better compression rate (and you really do ought to warn users that you're doing it by messing with critical OS files).

2 & 3) There's nothing wrong with what I've stated here. I do acknowledge in SSD speedup in #4, but for obvious reasons there's no way in hell I'll be NTFS compression any of my SSDs. The HDD backing my VM disk image is a 10k rpm velociraptor. I plan on running a single-threaded DrivePress later today to compare with the 2-thread version.

4) first, again my problem with compression on an SSD isn't the speed hit caused by fragmentation (it's several scores lower than the speed hit on a HDD, but it's still real) - it's (to some degree) the reduced speed and hindered wear-leveling on (at least, but probably not limited) drives with SandForce controllers, and (to a fairly large degree) the heavily increased amount of block erases caused by how NTFS compression is implemented. Having NTFs compression on often-modified files approaches suicidal tendencies for an SSD.

Because an SSD can read from/write to all parts of the drive at the same time (think of a hard disk platter rotating at infinite speed), that is why fragmentation is of absolutely no consequence for SSDs - be it NTFS compression induced, or the "normal" fragmentation that happens on NTFS inevitably. There is no delay, because all areas of disk are equally accessible at all times.-simonking (January 15, 2013, 04:13 AM)
--- End quote ---
This is patently wrong - take a look at some benchmarks. For instance, the 120GB Intel 510 drive does ~50MB/s for 4k random reads, whereas it does ~380MB/s for 128kb sequential reads (4k sequential would be slower, but should still be quite a lot faster than the random reads). You'll notice that it does 4k random *writes* faster, which is obviously because the drive has internal cache and can do (sequence-optimized) writes at leisure - and some of the other drives handle this even better.

a. Do NOT attempt to manually acquire file permissions just to be able to compress them. Doing this will create a huge security hole on your system (one that MagicRAR Drive Press does not create, because it restores all permissions as has been confirmed in this third party report).-simonking (January 15, 2013, 04:13 AM)
--- End quote ---
This is very good indeed - the way you handle the permissions is something to give credit for. I didn't look too closely at the code, but it seemed like you even throw in exception handling to do the permission-restore? You do leave files potentially vulnerable during the compression process... not much of a real-world problem, but could be reason enough for Microsoft consciously choosing not to do it. I still feel it wrong to classify the behaviour as a bug.

b. There will always be some files/folders that would be locked by the system/applications, and as such incompressible. If there is demand for it, we could also automate the conversion of those parts by building a boot time version of MagicRAR Drive Press - however, in my research, the additional space savings would be negligible.-simonking (January 15, 2013, 04:13 AM)
--- End quote ---
Agreed.

So while the MagicRAR Drive Press Challenge technically remains unmet-simonking (January 15, 2013, 04:13 AM)
--- End quote ---
Ho, humm, you still chose not to properly address any of the points of the original thread, most of which I showed to be clearly true. As I see it, only the points regarding interactions with SSD speed/lifetime can be debated... and for those points, I indeed do believe that I'm correct; what can be debated is to which degree lifetime and performance will be affected. For the current crop of SSDs, I definitely wouldn't do gung-ho NTFS compression, and I would recommend people against it.

Selective compression of static files would be fine, though. I wonder if it would make sense to apply compression on the files on another (and preferably HDD) partition, then move the files to the SSD target? I haven't tested, but it might result in less fragmentation of the target files.

Oh, and one last thing: your progress bars are severely bugged - they reached 100% several minutes before the actual operation was done (bugged both in analyze as well as compress phase). Looks like you use Delphi, and I haven't touched that since Delphi2, so dunno if there's limits on it's current/max values... but iirc the win32 controls are/were clamped to pretty low values, meaning you definitely shouldn't be using currentBytes/maxBytes - or even currentNumFiles/maxNumFiles for modern filesystems.

wraith808:
Oh, and one last thing: your progress bars are severely bugged - they reached 100% several minutes before the actual operation was done (bugged both in analyze as well as compress phase). Looks like you use Delphi, and I haven't touched that since Delphi2, so dunno if there's limits on it's current/max values... but iirc the win32 controls are/were clamped to pretty low values, meaning you definitely shouldn't be using currentBytes/maxBytes - or even currentNumFiles/maxNumFiles for modern filesystems.
-f0dder (January 15, 2013, 07:27 AM)
--- End quote ---

Could also be C++ builder.  And yes, there are limits for the reason you stated.  It's an int (16 or 32-bit depending on the version of comctrl32.dll [ref].

f0dder:
Could also be C++ builder.-wraith808 (January 15, 2013, 07:40 AM)
--- End quote ---
Ah yes, that can use the VCL (and other Delphi components) as well - didn't look closely, just saw some .pas references.

And yes, there are limits for the reason you stated.  It's an int (16 or 32-bit depending on the version of comctrl32.dll [ref].-wraith808 (January 15, 2013, 07:40 AM)
--- End quote ---
That reference mentions 64k limit - I wonder if comctrl uses signed or unsigned integers? It's been ages, but I seem to recall doing 32k clamping?

Navigation

[0] Message Index

[#] Next page

[*] Previous page