md5 / sha1 hashes What's the point?

[justice]

I sometimes see md5/sha1 hashes available to check programs/isos against. What is the point of them?

I know they can be used to verify that the program/software was downloaded without crc errors, but any good installer checks its contents for this before installing, and I've not had any crc errors for years in downloads. Hashes supposedly can tell you if the file has been tampered with, as I read the other day where an author's downloads were manipulated and they said: Always check your hashes. However surely it's trivial to change the webpage where the hash is listed or the hash file itself? How do you know that the person that made the installer and the hash file are the same person and are in fact the correct author of the download? So in my view it's not any help in this area either.

I must be missing something, so I ask: what is a good use of these hashes?

[worstje]

Hashes are a poor mans tampering/correctness test, as you said. It is mostly useful when the downloads are located on a different system than the actual announcement place.

If you want real security, you sign the package to get cryptographic blahblah on your side. Or maybe you can just sign the hash. I'm unsure, to be honest.

But yeah, the real use for them has eluded me too. Automated integrity tests seem like the only use to me.

[mouser]

I think they serve several purposes.

First, your point is correct, that if someone is hosting files on a web server and serving web pages from the same server, someone who could modify the files for download could also modify the displayed hash values.

However, in this case the author and others would presumably NOTICE that the web page hash values had suddenly changed.  So it would greatly increase the chance that the manipulation was noticed.

But more importantly -- it's quite common for large companies that use these kinds of hashes (Microsoft, etc.) to post the hash values for a download in multiple places on multiple servers.  And that's where the real security comes from.  Because it means that it's not practical for someone to manipulate the previously released distributed information about hash values.

Even if you are just an independent developer, you could simply post the hash values of an important file on a few forums, etc.  That would make it impractical for anyone to be able to manipulate those public hash values.  

In security lingo, this general concept of having a secondary channel of information that won't be vulnerable even if the main channel is tampered with is known as: out-of-band.

[worstje]

However, in this case the author and others would presumably NOTICE that the web page hash values had suddenly changed.  So it would greatly increase the chance that the manipulation was noticed.

-mouser (July 20, 2011, 03:44 AM)

It would be the 'others' more-so than the author, and not because the hash value changed but because the software acts up. As you say in the paragraph following that one, the original hash being posted as multiple locations is where the safety comes from - not from people mysteriously noticing a random string of hexadecimal digits changed.

You most likely can't even tell the first letter of the hash of the current release of some program, even if it were the only one you offer. Hell, if donationcoder.com got hacked and someone put an innocent <!-- mouser is a mouserer! --> in the source of a specific page - would you notice?

[justice]

Your point makes sense mouser, and this is the first time I heard about posting hash files in multiple places. Who ever checks that hashes are identical in multiple locations though before checking the download hash matches it? If that is what is required then I think the practicality of someone correctly using hashes is close to zero. How does someone know the other locations that store the hash file? A hacker could simple change the link to another site it controls, so now you need to verify the locations as well as the hash files. *head explodes*

[mouser]

not from people mysteriously noticing a random string of hexadecimal digits changed

All it takes is one person with an automated website-change watching program to easily and instantly spot (or be notified) when there is a change on a web page. Quite a few people run such tools (including me).

[mouser]

Who ever checks that hashes are identical in multiple locations though before checking the download hash matches it?

I don't, but some people do.

And really I think the main point is not to prevent the possibility that someone could get fooled into downloading a maliciously modified file.  The point is to set up a kind of early warning system so that it wouldn't go unnoticed if someone were tampering with files, and a way for people who NEED to make sure they have the official release to verify this.

You're never going to be able to prevent a malicious attacker from tricking a few people to downloading a trojaned file.  But you can make it fairly easy to rapidly detect when such attacks are attempted on trusted distribution sites, etc.

[Ath]

We also use the md5 to verify ftp downloads on unix as that sometimes messes up downloads.

[Ampa]

I use MD5 hashes to quickly run suspicious files through the VirusTotal system... it allows you to see results from their database for a specific file that has been tested in the past, without having to upload the file yourself.

[skwire]

I use hash files (mainly SFV files) to verify that my media collections do not get corrupted over time.  They also are great for verifying files after moving them from one computer/drive to another.

[f0dder]

I use hash files (mainly SFV files) to verify that my media collections do not get corrupted over time.  They also are great for verifying files after moving them from one computer/drive to another.

-skwire (July 20, 2011, 10:14 AM)

I've been doing that too - but lately I've been pondering if it wouldn't be a better idea to use PAR files instead... then you might be able to fix corruption rather than just detect it.

As for anti-tampering, this has already been touched on, but
for linux/bsd distros, the .iso files are hosted on a lot of different mirrors, and the links + hashes are hosted on the main site. To successfully violate a distribution, you'd need to hack both the main site as well as a number of mirrors (people downloading these things wouldn't be fooled by bad links). Some distributions even cryptographically sign the images.

If you want real security, you sign the package to get cryptographic blahblah on your side. Or maybe you can just sign the hash. I'm unsure, to be honest.

-worstje (July 20, 2011, 03:34 AM)

You're always signing a hash, as it'd be computationally unfeasible running public-key algorithms on a DVD image... whether you're signing the hash of the md5sum file or the hash of the DVD-image is a different matter, though :p

but any good installer checks its contents for this before installing

-justice (July 20, 2011, 03:26 AM)

Installers work differently, though. For OS distributions, you don't generally check the entire media before installing - especialling considering you're often running from an optical media, that would be extremely slow... and considering you don't install 100% of the packages on the media, it would also be stupid. Better to let users offline-verify the ISO hash, and then online-verify the individual packages being installed.