Author Topic: Partial (corrupted) downloads from a server (Not a valid win32 application etc) (Read 35,915 times)

mouser · « **on:** April 11, 2011, 01:05 PM »

We've always had an occasional occurrence of this problem, but it seems to be getting worse lately.. Someone will try to download a program from our website (main or member server), and when they try to run it, they get an error saying its "not a valid win32 application" or "The setup files are corrupted".

Worse yet, if they try to re-download the browser gives them the cached half-finished download again so the problem recurs and they are left very confused and thinking the software is defective.

A search on the web shows that this does happen enough to everyone that the problem is well known

But my question is: Does anyone have any server experience who can help us understand how to fix this problem on the server end -- to stop this from happening so often? What is causing downloads to fail half way through, and leave the browser thinking the download is complete? And how can we fix this on our server end?

Does anyone have any server running experience have any idea? Help!

vlastimil · « **Reply #1 on:** April 11, 2011, 01:42 PM »

Not that I know the answer for that question (as much as I would like to know) (maybe fiddling with http headers would help with the re-download), but it also sometimes happens to me and I usually have the download on multiple urls and have an additional .zip with the installer and I tell the people to download using one of the other urls or with different browser.

Stoic Joker · « **Reply #2 on:** April 11, 2011, 04:55 PM »

Does the server allow download resuming?

Gothi[c] · « **Reply #3 on:** April 11, 2011, 09:17 PM »

Quote from: Stoic Joker on April 11, 2011, 04:55 PM

Does the server allow download resuming?

Yes. (Apache does by default)

mouser · « **Reply #4 on:** April 11, 2011, 09:24 PM »

The thing i find strangest is that the browser is not figuring out that the download is incomplete.. that's what's really throwing me for a loop -- i mean normally the browser knows when a connection has failed, etc. What's particularly bizarre in this case is that the browser and server are somehow convinced that the download has fully finished successfully.

Ath · « **Reply #5 on:** April 11, 2011, 09:36 PM »

Are you losing any network packets between the server and the nearest measurable connection, like a switch that went goofy or something like that (aka: [partial] hardware failure)?

mouser · « **Reply #6 on:** April 11, 2011, 09:40 PM »

loss of packets is quite possible -- but normally in this case doesn't the browser detect that it is having communication problems and stall, etc. I mean thats what i see happen when we know there is a packet loss issue. The result is slow downloads not corrupted ones.

worstje · « **Reply #7 on:** April 11, 2011, 09:57 PM »

Broken hardware is my bet. Several years ago, in a bit of a convoluted setup involving a big room, several bookcases and a troublesome place where the wire came out of the wall, I had a router that balanced on a bunch of heavy law books. And when cleaning, someone apparently made it fall off those books before putting it back as if nothing had happened.

The results were extra-ordinary: rather than have the network connection die or anything of the sort, stuff just started getting garbled. Websites had garbled images with random lines and color shifts and the sorts; HTML documents had random characters happen and so forth. So somehow this entire affair happened somewhere inbetween the layers of error-correction; I suspect the router had its memory or cpu get messed up, but had its networking interfaces still work peachy.

Perhaps your situation is similar in nature here.

JavaJones · « **Reply #8 on:** April 11, 2011, 10:01 PM »

I kind of don't think such an unusual situation as worstje describes is at play. I say that mostly because I've seen similar issues many times from other servers. I personally think it's a failure of the browser's caching mechanisms or something user-side, not server-side (not certain though). Granted it doesn't happen for all downloads, which might suggest it is a server-specific thing, but when it does happen it does seem to (usually) help to clear the browser's cache and download history, so that says something to me...

- Oshyan

mouser · « **Reply #9 on:** April 11, 2011, 11:07 PM »

Its not limited to a specific browser brand -- it happens in all browsers. It's random. Its not a corrupted byte, it's a download that stops somewhere in the middle but ends up with the browser thinking it has successfully downloaded the entire file. And yes, at that point the browser is so convinced that it has successfully downloaded the full file, that it will refuse to redownload it, and will serve up the cached copy when the user tries to redownload -- making the problem much, much worse and more confusing for the user.

jaden · « **Reply #10 on:** April 12, 2011, 05:26 PM »

A few months ago I ran into a problem with MediaTemple where certain downloads were corrupted. It turned out the network guys had enabled a virus scanner that was incorrectly flagging certain files as having a virus. They admitted the scanner had never correctly identified a payload, only given false positives, and disabled it so our downloads worked. Perhaps something along those lines is happening here?

jaden · « **Reply #11 on:** April 12, 2011, 05:28 PM »

It could also be an anti-virus app on the user's system that's scanning downloads and messing with the file. Do the failures occur with particular files, or is it completely random?

mouser · « **Reply #12 on:** April 12, 2011, 05:39 PM »

it's usually the bigger exe setup files i think.. but i'm not sure there is a real pattern to it other than that the files are usually several megabytes large.

JavaJones · « **Reply #13 on:** April 12, 2011, 05:41 PM »

Having experienced this myself on the client side I can say with fair certainty it's not likely to be antivirus-related on the client end (but maybe server side AV?). The reason I say this is that I've had it happen across a range of different AV software and configuration options, including situations where the scanning of web downloads is not automatically enabled in the AV but is instead called by the browser once the download is finished (suggesting again that the browser is prematurely concluding the file is complete).

- Oshyan

f0dder · « **Reply #14 on:** April 12, 2011, 05:50 PM »

Not sure if it's in any way related, but FWIW I can't reliably grab the minecraft picture from this dcmember - displays a bit, and then stalls (FireFox 4). With wget, I'm seeing download rates at less than 10kb/s, and then massive stall. These images are relatively large, probably around a couple megabytes for some of them.

Apart from Microsoft Security Essentials, I'm not running any anti-malware or internet filtering junk - and the situation isn't any better when wget'ing from my linux server.

jaden · « **Reply #15 on:** April 12, 2011, 05:59 PM »

Quote from: f0dder on April 12, 2011, 05:50 PM

Apart from Microsoft Security Essentials, I'm not running any anti-malware or internet filtering junk - and the situation isn't any better when wget'ing from my linux server.

That pretty much rules out an issue on the client side if wget is failing. In the MediaTemple case it only happened with very large files (more chance to conflict with a known virus pattern?) but filtering only occurred when the files were served from Apache. Can someone scp a problem file to another server to see if the file comes across alright?

jaden · « **Reply #16 on:** April 12, 2011, 06:08 PM »

I just tried downloading this PNG: http://carrolld.dcmembers.com/minecraft/2011-04-07.png and I saw the behavior being described (browser thought it was done, but only the top portion of the downloaded image is visible).

In our case the files were zips and were corrupted, so I'm not sure if this could be the same issue or not. It's possible when a server-side scanner flags a file as infected it stops the download and the browser thinks it's complete.

JavaJones · « **Reply #17 on:** April 12, 2011, 06:10 PM »

Great find f0dder, that seems likely related. What's interesting is that the download seems to start out quickly then slow down terribly (the same behavior is common to a browser and wget with those specific PNG files). Fetching a similarly sized EXE file (FlipSuite in my test) went like lightning.

- Oshyan

40hz · « **Reply #18 on:** April 12, 2011, 06:26 PM »

Wonder if this is somehow related to the stalled picture upload problem you'll occasionally encounter during forum posting. If so, then it's definitely a server rather than a client-side issue.

I seem to recall reading somewhere that there was an issue with certain php download scripts that retrieved the requested file from a mysql database and were responsible for corrupted downloads. Had something to do with buffer settings iirc.

Can't find it now of course. Anybody else remember seeing that?

jaden · « **Reply #19 on:** April 12, 2011, 06:31 PM »

Another data point. The initial download of http://carrolld.dcmembers.com/minecraft/2011-04-07.png in Chrome produced a 24KB file. In Firefox it stalled at 156KB and Opera stalled at 209KB.

Stoic Joker · « **Reply #20 on:** April 12, 2011, 08:24 PM »

Quote from: jaden on April 12, 2011, 06:31 PM

Another data point. The initial download of http://carrolld.dcmembers.com/minecraft/2011-04-07.png in Chrome produced a 24KB file. In Firefox it stalled at 156KB and Opera stalled at 209KB.

I may have just got lucky, but IE9 grabbed that file and all the others at the link f0dder posted just fine on the first shot.

Stoic Joker · « **Reply #21 on:** April 12, 2011, 08:27 PM »

Quote from: 40hz on April 12, 2011, 06:26 PM

I seem to recall reading somewhere that there was an issue with certain php download scripts that retrieved the requested file from a mysql database and were responsible for corrupted downloads. Had something to do with buffer settings iirc.

That does sound familiar, but I was thinking it was only an ASP thing (but I could be wrong). Had to do with disabling the caching if script execution results and just spitting it up the wire ... Yes??

mwb1100 · « **Reply #22 on:** April 12, 2011, 08:57 PM »

I don't know if this will help anyone narrow this down (or just add noise to the data), but I often have trouble downloading large files in Firefox or IE at work (no errors reported, just incomplete files after the download finishes). When that happens I use wget which has yet to have a problem.

That's an indication to me that the problem is the browser's fault, but what do I know?

Anyway, if someone here has these issues downloading in the browser, it might shed some light here if they find whether or not wget is more reliable.

jaden · « **Reply #23 on:** April 12, 2011, 10:00 PM »

Quote from: Stoic Joker on April 12, 2011, 08:24 PM

I may have just got lucky, but IE9 grabbed that file and all the others at the link f0dder posted just fine on the first shot.

I tried it again and it downloaded completely in all 3 browsers, as well as wget.

Ath · « **Reply #24 on:** April 12, 2011, 10:16 PM »

Quote from: jaden on April 12, 2011, 10:00 PM

I tried it again and it downloaded completely in all 3 browsers, as well as wget.

I could get it to load completely in FF3 just now for the first time too

[wishful thinking mode]
Usually that happens after mouser and Gothi[c] do some config changes to the/a server, they wait for a little time to see if 'that helped', and if it did they happily report it fixed here

Let's just wait for that then

(I hope so, keeping my fingers crossed)
[/wishful thinking mode]