Does KGB Archiver really achieve high compression rates?

[Eóin]

I came across KGB Archiver a while back and really don't know what to make of it.

KGB Archiver is the compression tool with an unbelievably high compression rate. Unfortunately, in spite of its powerful compression rate, it has high hardware requirements (I recommend processor with 1,5GHz clock and 256MB of RAM as an essential minimum). One of the advantages of KGB Archiver is also AES-256 encryption which is used to encrypt the archives. This is one of the strongest encryptions known for human.

-http://kgbarchiver.net/

Initially I ran a simple test against 7 zip on a small source bundle, it failed miserably but I hadn't used to highest compression so didn't want to discount it nor can I remember to exact figures to post here. Now I have run a second test and it once again it seriously disappoints.

I compressed a VirtualBox VDI (virtual disk image) file of a Server 2008 install. That's 5.8 GB uncompressed. I think these figures speak for themselves.

7 zip (ultra compression)	984 MB	~45mins
KGB Archiver (highest settings)	1.19GB	~16hrs 30mins

Is it all a scam or have I just happened to pick two very difficult test cases?

[mwb1100]

I haven't tried KGB (or even downloaded it), but it seems harsh to even remotely call it a 'scam'.

It's an open-source project that makes some impressive claims about it's compression ratios.  While it might not work as well in your tests as the claims suggest - indeed it may not even be suitable for use (16 hours is a long time to spend compressing) - I think that applying the term 'scam' to software should be reserved for things involving malware or fraud. It should not be used in a situation where a compression utility doesn't perform quite as well as another.

[Renegade]

It's using different PAQ flavours for compression...

They are NOT CPU friendly in the least. They are extremely slow, but they do get better compression than pretty much anything else (in general that is). Check out Maximum Compression. You'll see that PAQ compression schemes are always among the top.

PAQ is inefficient and at maximum settings, almost unusable for anything but very small archives. As you have seen with a 16+ hour go at it for 5.8GB of data. Not fun.

Efficiency becomes more important (for humans) with larger archives.

[Renegade]

Check out the multi-file compression test at Maximum Compression.

PAQ8O8 at the top takes 43,660 seconds. That's 12 hours, 7 minutes, 40 seconds. Scroll down and you'll see ALZip does ALZ compression in 36 seconds. BIG difference.

Then again, ALZ gets 63% compression, while PAQ8O8 gets 80% -- 17% more, but at a cost of an additional 12 hours, 7 minutes, 4 seconds.

Different formats have different strengths. It's best to know what you're doing when you choose a format. If not, just go for ZIP compression.

[Eóin]

I used the word scam because I heard about it on forums with claims of near 1% compression ratios. But in fairness those claims were being made by others, it's own website doesn't make such a claim so I've changed the title of the thread as to not imply deceit on the developers part.

I can imagine an excellent compression ratio even at the expense of seriously heavy cpu costs could be justified if the compression is a one time cost. Say in distributing large downloads, the bandwidth saving could justify the initial compression investment.

But given that in both my tests it underperformed 7 zip even before taking time into account I wondered was it not all it seemed to be

[f0dder]

Slow compression can be justified the compression rate is very good, and that decompression is substantially faster...

[tinjaw]

A think something that many people overlook is that many of today's data files are already compressed and further compression is going to take time and CPU cycles. For example mp3, jpg, gif, jar, and others are already compressed.

[f0dder]

.jar are just .zip files - for download (as opposed to applet-on-a-site) use, I'd probably use STORE compression method, so the .jar's can be properly compressed

But yes, you do have a point, lots of file formats don't compress very well.

[tinjaw]

Here is something I have been meaning to research. I assume programs like 7-zip, WinRAR and the like must have an option to take into consideration the file being compressed. I assume you could have a special compression setting that compresses stuff in different ways depending on the source file.

It might even be advantageous to take an look at the files beforehand and create an archive, putting the highly compressible stuff in front and the already compressed stuff in the back. It could then switch algorithms as it goes, creating chunks, text and stuff using one algorithm and mp3, jpg, jar, using another. Or does somebody already do this?

[f0dder]

WinRAR sorts input files based on extension, and there's an additional "RarFiles.lst" so similar-in-content files are grouped together: ie., .html and .txt would be relatively "far away" if grouped just on extension, but they have similar (plain-text) format, and thus should be grouped together-ish. The file also has a "$default" entry where the group-by-extension logic goes, and after that basically uncompressable files are listed so they don't "poison" the compression dictionary.

Also, it has supported "input filtering" for a while, which basically makes some (100% reversible) translations on certain input formats (iirc .wav, .bmp and .exe are included) to achieve better compression for those formats.

[Lashiec]

I used the word scam because I heard about it on forums with claims of near 1% compression ratios. But in fairness those claims were being made by others, it's own website doesn't make such a claim so I've changed the title of the thread as to not imply deceit on the developers part.

-Eóin (March 24, 2008, 04:29 AM)

Yeah, I read that somewhere recently and I was all like "WTF?". It's hard to go beyond current compression ratios, and with modern technology (and specialized algorithms for certain data like tinjaw mentions), it's not worth the work of creating a new scheme, and waste tons of CPU time just to squeeze a few megabytes.

I remember that a friend showed me a couple of years ago a super-compressed package of data whose creators claimed impressive ratios, but at the cost of really long decompression times (which I witnessed first hand). Now, if I could remember where did it come from...

[nosh]

You could compress something like 1 GB to under 5 KB, it all depends on what you're compressing.
I'd upload the file but I'm scared some forum AV might try to peek into it.

[Carol Haynes]

I compressed a VirtualBox VDI (virtual disk image) file of a Server 2008 install. That's 5.8 GB uncompressed.

-Eóin (March 23, 2008, 07:44 PM)

Ah but is it uncompressed? If you are downloading a virtual disc there is an emphasis on the image being as compacted as possible so have the builders of the image turned on disc compression in Windows when they were packaging it up? Also there are loads of CAB files which are already compressed.

Having said that if 7-zip can do a better job in 45 minutes (not exactly known for its speed either) then what is the point of KGB ?

[Eóin]

Having said that if 7-zip can do a better job in 45 minutes (not exactly known for its speed either) then what is the point of KGB ?

-Carol Haynes (March 24, 2008, 08:03 PM)

That's really the only comparison I was making. I gave it the random test to see how much it would beat 7-zip by so I could then make a subjective decision as to if the extra time needed could still be justified in certain cases. The compression ratio itself I wasn't worried about because as already pointed out that depends very much on the data.

That it produced bigger archives than 7-zip really surprised me and left me bewildered

[Renegade]

Don't forget that content matters. A lot. Different compression schemes will work better with different types of files. Check Maximum Compression and have a look. It's a bit dense as there are so many there, but you'll see that pan out.

The 4 basic areas for compression algorithms are:

1) Audio
2) Video
3) Images
4) Data (always lossless)

The first 3 have their own specific algorithms. e.g. AC3 compression, MPEG-2 compression, JPEG compression. Most data compression schemes will not yield significant improvements over those. PAQ will usually, but at a high CPU cost.