Utility to get one good copy from three corupted ones?

[Eóin]

Hey all, these last two days I've been trying to download the recent MSDN Library (April 2007 Edition). Three times I've downloaded it and each time I've gotten three different hashes none of which match the ones from the download page.

Instructions
...snip...
Large Download size
The size of this download is over 2 GB and due to the large size, there have been reports of file corruption during file download. It is recommended to run a CRC verification after downloading.

SHA1 hash: 0x1ACA38B900F89049457EDC325198DC8419C4ACE1.
Or CRC value: 0x24BFED4A.

-www.microsoft.com/downloads

These are the hashes I get

1st
SHA-160     : F53D30759AE25C3DC32109444CBFEC9612A5FA0A
CRC-32      : B69C368F

2nd
SHA-160     : 5BA5BB78DEBD6BF27616AD9BFF420F6D9F8375F2
CRC-32      : 8C28E28C

3rd
SHA-160     : 4074EA3D7F095B466FD8424E24F8029239B14D92
CRC-32      : F6977EEA

Assuming the corruptions are small and different in each file it should be possible to construct a correct file from the three I have. Anyone ever come across a utility that might do something like that? It's frustrating because I've never had problems like this from the p2p networks

[compn]

if there is a torrent of the original file, you could join the torrent and select your bad hash file as a partial file. then the torrent program would scan your file and redownload the bad chunks...  (and since you have three files, you would load up all of them , perhapse in three different clients or pc's? ,and then join the torrent, it might be that you have a complete file out of your three downloads)

this also would work on the edonkey p2p network , i think...

also there is a freeware program called 'zidrav'. so if you know of someone else that downloaded the file, you both run zidrav and you send him the checksum and he sends you a 'patch'

ZIDRAV is a file corruption detection and repair program. It's designed to provide a way to fix file damage without having to redownload the entire (potentially large) file over a (potentially slow) connection.

http://sourceforge.net/projects/zidrav

-- i bet this program could be advanced with a nice gui and perhapse some internet support/automation, if any programmers want to take a crack at it--


but yeah, interesting problem and good luck!

[Hirudin]

if there is a torrent of the original file...

-compn (May 09, 2007, 10:18 PM)

That sounds like a good idea.

You could also split all 3 files into small pieces (make copies first of course), maybe 2000 1mb pieces*. Then compare set A against set B using something like beyond compare. Copy one set of files that were tested to be exactly the same to folder D. Then compare A to C, copy another set of duplicates to folder E. Then compare B to C, copy one last set to F. Compare D to E, all the files that are in both D and E should be exactly the same with a few "orphan" files here and there. Sync the 2 folders so you have all the same files in both D and E, then delete one of them, say E. Then compare D to F, sync them, and delete one, say F.

Folder D should have one complete set of all the files, the best bet for accuracy since every file in it got downloaded at least 2 times exactly the same way.

Unsplit the files and check the MD5 again, hopefully it'll be correct.

If folder D is missing any files, I guess you're stuck with doing compn's idea (which might be easiest if you already have uTorrent or something and there is a torrent available) or I guess you could stick the missing files in from folders A, B, and C in a systematic manner (might get pretty complicated if there's more than like, 3 missing files).

*Although I haven't actually used this feature, I think Directory Opus will split files, if you don't have it there are tons of file splitter programs out there. I don't remember the reasons why now, but of the programs I've tried "The File Splitter" is the only one I've kept.

[Eóin]

Thanks for the suggestions guys!

compn, looking for a bittorrent or emule copy was a very good idea, unfortunately I couldn't find either. I looked into zidrav too. It's a neat program, one to remember, but I don't know anyone with a clean copy to get a patch from.

Hirudin, I was thinking along the same lines as you too. I decided to write a tiny app to scan through the three copies byte by byte and any time a discrepancy showed up in just one copy it would discard it.

That threw up some interesting results. First there were loads of discrepancies, so many that you'd end up wondering how anything over 100MB ever downloads correctly. Still, things looked good initially as there was never a spot where all three files disagreed. In the end though the new cleaned copy produced just gave another 4th hash still not matching the MS ones.

So, I'm going to try one last time here on the college connection using IE7 for a change. If that doesn't work I'll just get it myself off of usenet over the weekend. I saw a posted copy there with 300MB extra of par2 redundancy files. I'll have to get a clean version from that.

Thanks again for the suggestions

[justice]

I'm downloading the file, should be done in 40 min, and wil use that sourceforge.net software if you want.

I guess it would work if you used .par / par2 files and then repair the file using quickpar.

[Eóin]

Hi justice, with ZIDRAV as I understand it I'd need to create a hashfile with my corrupted version and send that to you from which you could create a patch file. Unfortunately I can't do that today because I'm away from the computer with the previous downloads. And the 4th copy I was going to download again today in college I can't seem to get at all. I'm definitely thinking usenet over the weekend is the best option.

The funny thing is I don't even need the file, I have use for it of course, but it's not crucial to anything either. I think at this stage I just don't want it to beat me

[justice]

You've been using the wrong hash program I think. you need the SHA1 hash!
I use DPASHA which does the SHA1 hash, it downloaded correctly with FlashGet

I'll enclose a par2 file. Tell me how many blocks you need...

[Eóin]

Hi justice, thanks when I get back to the other copies I'll I'll check with the par2 files. I'd been using Damn Hash Calculator for the hashes, it's old but I had it lying around and it worked for me in the past, it's hash values matched with DPASHA.

Is FlashGet adware free these days? I used to use it back two or three years ago, stuck to an early version with the bundled adware removed. Only moved on from it when I started using the DownThemAll extension to FireFox which seemed to do just as good a job.

[justice]

Yep FlashGet has gone freeware, and it's still as good. It's listed on softpedia as 100% no spyware / adware.

I've uploaded the par files: http://www.box.net/shared/cbypmb43x1

[Eóin]

Ok, this is one of those things which s so bad it's funny, there was obviously something interfering with this download because it's nearly completely corrupted

[attach#]

[justice]

I only uploaded 7 recovery blocks which was 49MB, looks like you best download it from scratch indeed, the patch / pars would be as big as the original.