DonationCoder.com Forum
Other Software => Developer's Corner => Topic started by: mouser on April 11, 2011, 08:05 AM
-
We've always had an occasional occurrence of this problem, but it seems to be getting worse lately.. Someone will try to download a program from our website (main or member server), and when they try to run it, they get an error saying its "not a valid win32 application" or "The setup files are corrupted".
Worse yet, if they try to re-download the browser gives them the cached half-finished download again so the problem recurs and they are left very confused and thinking the software is defective.
A search on the web shows that this does happen enough to everyone that the problem is well known
But my question is: Does anyone have any server experience who can help us understand how to fix this problem on the server end -- to stop this from happening so often? What is causing downloads to fail half way through, and leave the browser thinking the download is complete? And how can we fix this on our server end?
Does anyone have any server running experience have any idea? Help!
-
Not that I know the answer for that question (as much as I would like to know) (maybe fiddling with http headers would help with the re-download), but it also sometimes happens to me and I usually have the download on multiple urls and have an additional .zip with the installer and I tell the people to download using one of the other urls or with different browser.
-
Does the server allow download resuming?
-
Does the server allow download resuming?
-Stoic Joker
Yes. (Apache does by default)
-
The thing i find strangest is that the browser is not figuring out that the download is incomplete.. that's what's really throwing me for a loop -- i mean normally the browser knows when a connection has failed, etc. What's particularly bizarre in this case is that the browser and server are somehow convinced that the download has fully finished successfully.
-
Are you losing any network packets between the server and the nearest measurable connection, like a switch that went goofy or something like that (aka: [partial] hardware failure)?
-
loss of packets is quite possible -- but normally in this case doesn't the browser detect that it is having communication problems and stall, etc. I mean thats what i see happen when we know there is a packet loss issue. The result is slow downloads not corrupted ones.
-
Broken hardware is my bet. Several years ago, in a bit of a convoluted setup involving a big room, several bookcases and a troublesome place where the wire came out of the wall, I had a router that balanced on a bunch of heavy law books. And when cleaning, someone apparently made it fall off those books before putting it back as if nothing had happened.
The results were extra-ordinary: rather than have the network connection die or anything of the sort, stuff just started getting garbled. Websites had garbled images with random lines and color shifts and the sorts; HTML documents had random characters happen and so forth. So somehow this entire affair happened somewhere inbetween the layers of error-correction; I suspect the router had its memory or cpu get messed up, but had its networking interfaces still work peachy.
Perhaps your situation is similar in nature here.
-
I kind of don't think such an unusual situation as worstje describes is at play. I say that mostly because I've seen similar issues many times from other servers. I personally think it's a failure of the browser's caching mechanisms or something user-side, not server-side (not certain though). Granted it doesn't happen for all downloads, which might suggest it is a server-specific thing, but when it does happen it does seem to (usually) help to clear the browser's cache and download history, so that says something to me...
- Oshyan
-
Its not limited to a specific browser brand -- it happens in all browsers. It's random. Its not a corrupted byte, it's a download that stops somewhere in the middle but ends up with the browser thinking it has successfully downloaded the entire file. And yes, at that point the browser is so convinced that it has successfully downloaded the full file, that it will refuse to redownload it, and will serve up the cached copy when the user tries to redownload -- making the problem much, much worse and more confusing for the user.
-
A few months ago I ran into a problem with MediaTemple where certain downloads were corrupted. It turned out the network guys had enabled a virus scanner that was incorrectly flagging certain files as having a virus. They admitted the scanner had never correctly identified a payload, only given false positives, and disabled it so our downloads worked. Perhaps something along those lines is happening here?
-
It could also be an anti-virus app on the user's system that's scanning downloads and messing with the file. Do the failures occur with particular files, or is it completely random?
-
it's usually the bigger exe setup files i think.. but i'm not sure there is a real pattern to it other than that the files are usually several megabytes large.
-
Having experienced this myself on the client side I can say with fair certainty it's not likely to be antivirus-related on the client end (but maybe server side AV?). The reason I say this is that I've had it happen across a range of different AV software and configuration options, including situations where the scanning of web downloads is not automatically enabled in the AV but is instead called by the browser once the download is finished (suggesting again that the browser is prematurely concluding the file is complete).
- Oshyan
-
Not sure if it's in any way related, but FWIW I can't reliably grab the minecraft picture from this dcmember (http://carrolld.dcmembers.com/minecraft/) - displays a bit, and then stalls (FireFox 4). With wget, I'm seeing download rates at less than 10kb/s, and then massive stall. These images are relatively large, probably around a couple megabytes for some of them.
Apart from Microsoft Security Essentials, I'm not running any anti-malware or internet filtering junk - and the situation isn't any better when wget'ing from my linux server.
-
Apart from Microsoft Security Essentials, I'm not running any anti-malware or internet filtering junk - and the situation isn't any better when wget'ing from my linux server.
-f0dder
That pretty much rules out an issue on the client side if wget is failing. In the MediaTemple case it only happened with very large files (more chance to conflict with a known virus pattern?) but filtering only occurred when the files were served from Apache. Can someone scp a problem file to another server to see if the file comes across alright?
-
I just tried downloading this PNG: http://carrolld.dcmembers.com/minecraft/2011-04-07.png (http://carrolld.dcmembers.com/minecraft/2011-04-07.png) and I saw the behavior being described (browser thought it was done, but only the top portion of the downloaded image is visible).
In our case the files were zips and were corrupted, so I'm not sure if this could be the same issue or not. It's possible when a server-side scanner flags a file as infected it stops the download and the browser thinks it's complete.
-
Great find f0dder, that seems likely related. What's interesting is that the download seems to start out quickly then slow down terribly (the same behavior is common to a browser and wget with those specific PNG files). Fetching a similarly sized EXE file (FlipSuite in my test) went like lightning.
- Oshyan
-
Wonder if this is somehow related to the stalled picture upload problem you'll occasionally encounter during forum posting. If so, then it's definitely a server rather than a client-side issue.
I seem to recall reading somewhere that there was an issue with certain php download scripts that retrieved the requested file from a mysql database and were responsible for corrupted downloads. Had something to do with buffer settings iirc.
Can't find it now of course. Anybody else remember seeing that?
-
Another data point. The initial download of http://carrolld.dcmembers.com/minecraft/2011-04-07.png (http://carrolld.dcmembers.com/minecraft/2011-04-07.png) in Chrome produced a 24KB file. In Firefox it stalled at 156KB and Opera stalled at 209KB.
-
Another data point. The initial download of http://carrolld.dcmembers.com/minecraft/2011-04-07.png (http://carrolld.dcmembers.com/minecraft/2011-04-07.png) in Chrome produced a 24KB file. In Firefox it stalled at 156KB and Opera stalled at 209KB. -jaden
I may have just got lucky, but IE9 grabbed that file and all the others at the link f0dder posted just fine on the first shot.
-
I seem to recall reading somewhere that there was an issue with certain php download scripts that retrieved the requested file from a mysql database and were responsible for corrupted downloads. Had something to do with buffer settings iirc.-40hz
That does sound familiar, but I was thinking it was only an ASP thing (but I could be wrong). Had to do with disabling the caching if script execution results and just spitting it up the wire ... Yes??
-
I don't know if this will help anyone narrow this down (or just add noise to the data), but I often have trouble downloading large files in Firefox or IE at work (no errors reported, just incomplete files after the download finishes). When that happens I use wget which has yet to have a problem.
That's an indication to me that the problem is the browser's fault, but what do I know?
Anyway, if someone here has these issues downloading in the browser, it might shed some light here if they find whether or not wget is more reliable.
-
I may have just got lucky, but IE9 grabbed that file and all the others at the link f0dder posted just fine on the first shot.
-Stoic Joker
I tried it again and it downloaded completely in all 3 browsers, as well as wget.
-
I tried it again and it downloaded completely in all 3 browsers, as well as wget.
-jaden
I could get it to load completely in FF3 just now for the first time too
[wishful thinking mode]
Usually that happens after mouser and Gothi[c] do some config changes to the/a server, they wait for a little time to see if 'that helped', and if it did they happily report it fixed here :tellme:
Let's just wait for that then :-[ (I hope so, keeping my fingers crossed)
[/wishful thinking mode]
-
Usually that happens after mouser and Gothi[c] do some config changes to the/a server, they wait for a little time to see if 'that helped', and if it did they happily report it fixed here
-Ath
That was my first thought too, but I wasn't daring enough to move into full-on wishful thinking mode ;)
-
I may have just got lucky, but IE9 grabbed that file and all the others at the link f0dder posted just fine on the first shot.
-Stoic Joker
I tried it again and it downloaded completely in all 3 browsers, as well as wget.
-jaden
Great, We've confirmed the problem is intermittent. :wallbash:
:D
-
Just tried http://carrolld.dcmembers.com/minecraft/2011-04-07.png in IE8, Firefox 3.6.16, Chrome, Opera and Safari and it loaded in each with no problem.
I have to say I have come across this problem before with client machines that won't complete downloads but I have often found other machines download them fine - even on the same WiFi network.
It's strange and I have never found an acceptable answer to the problem.
Could it be something to do with faulty caching at ISPs when it is a more general problem?
-
Works here now too, FireFox as well as wget from the Linux box. Seems to have relatively slow start, but I hit 330kb/s for one of the larger images :Thmbsup:. Hope it's Gothic that's been sprinkling magic dust on the servers :)
-
So far this is what we have seen in this thread:
Initially, at least 3 people (f0dder, jaden, and myself) were able to confirm the issue with downloading those PNGs.
At least 2 confirmed that wget had the same problem during that time period.
Subsequently multiple people confirm that the issue is no longer present either with wget or the browser.
What this says to me is A: it's not browser-specific (wget had the same problem previously while the issue was occurring) and B: it's intermittent.
Hopefully all this is moot and they've found a fix already. :D
- Oshyan
-
Hopefully all this is moot and they've found a fix already
Nope. Let me clarify this -- We are at a loss as to how to fix this and what's causing it -- that's why I posted.
It's not a new phenomena but it seems to have gotten worse since the server move in January. It's obviously not just our server suffering from this, as a web search reveals.
But this is not something easy -- as you have noticed it's very intermittent and random and hard to reproduce.
-
And it isn't related to the datacenter, or it's connection to the backbone, where the servers are hosted?
-
it might be.. if someone could explain how or what could be wrong that could cause this..
i'll repeat the point i keep trying to emphasize -- the internet is a giant heap of malfunctioning junk powered by an infrastructure of software that is insanely retarded.. but even so, when downloading a file if you have connection problems, your browser knows that the download has not finished and waits there until the connection recovers, or else aborts and knows that it has failed with an incomplete file. and the communication protocols know when there is packet loss and request packets to be resent, etc.
the salient point about this mystery is that something is truly malfunctioning in this normal process which causes downloads to stop in the middle, and the browser is getting completely confused into thinking that the download has completed successfully.
the question is how can this happen -- in what specific way are things getting so fouled up as to result in the browser thinking it has successfully downloaded a complete file without error when it hasn't, bypassing all of the normal processes that normally detect and recover from bad transfers?
-
Can you temporarily disable download resuming to see what (if any) effect it has on the behavior?
Even if the server had a shaky connection to the backbone, the chances of router munging a packet to perfectly resemble transfer complete I just don't see happening.
Nobody jumped at the script timeout angle, and it doesn't appear that any are being used.
There are to many client side variations experiencing the same issue for the problem to exist on that end.
(Bear with me I'm just thinking out loud here. :))
What are the chances of a file system glitch making the server think the file is smaller than it really is?
-
Keep up the great ideas, suggestions, clues, etc. Definitely some good ideas on this thread and I know gothic (dc server admin) will be trying some of this stuff to see if it has any effect or triggers any epiphanies.
What are the chances of a file system glitch making the server think the file is smaller than it really is?
Seems unlikely to me -- if that was happening, i would think the server would be experiencing much graver catastrophic crashes in all kinds of operations.
-
mouser,
Just mentioning this in case it's related. I'm still having a problem attaching images. I just tried uploading a 230 KB image right now to confirm. "The connection to the server was reset while the page was loading." Got the error in under 10 seconds.
I don't have this problem with other sites so I don't think it has to do with my connection. Maybe it _is_ my connection and other sites are more tolerant about timeouts or something?
BTW, I loaded the above png fine, as expected. I haven't had any issues with pages timing out while surfing the forum.
-
Given that this was reproduced with direct file access to those PNGs, it doesn't seem like a "script timeout" issue as far as I can imagine.
Mouser reiterates the critical issue: it's not necessarily surprising that the transfers should break or whatever, it's the fact that when they do, the browser doesn't realize it and subsequent attempts to download the same file (from the same location) don't trigger a re-download but instead just use a cache.
This makes me wonder whether it may be some kind of in-between ISP caching. I know someone else mentioned that. Very hard to say indeed.
It's also important to note we may be looking at 2 separate but related issues. On one side certainly there are occasional/intermittent transfer issues, as seen with the PNG issue pointed out earlier in this thread. These would also be likely at the root of, or at least involved in, the reason the installer download problems are occurring. So that issue certainly seems to need investigation and resolution if possible, and it seems that can only be either server-side or at some link in-between since it was reproduced simultaneously on multiple distant systems with (presumably) different Internet providers. It would be interesting to test packet loss to the server from a machine that is currently experiencing the slow download issue. It's also important to note it *seems* to be isolated to particular files on the server when it occurs, which is an additional twist which really makes it confusing, but at least suggests that it is more specific to the server than an in-between link (maybe...).
And then on the other side is the issue of the browser not realizing - or not being properly told - that a download is actually not complete when the user requests to download it again. This seems to occur cross-browsers so it does not suggest an issue in a particular browser's caching mechanism. While there may be such an issue common to many browsers, it seems somewhat dubious given they all work somewhat differently (for example Firefox uses temp files in the same directory as the download, while IE uses the system's temp folder and then copies the finished file to the destination folder when complete). Another possibility is some kind of caching in-between, likely at the ISP level (I don't know if the higher-level interlink providers like Level 3 do caching though). This would explain why it happens cross-browser. However my own experience with this issue suggests in fact that is not always so consistent. Sometimes using another browser *will* help. So it's all very confusing.
Bottom line I think the intermittent transfer speed issues from the server are a clear problem and need to be resolved. If they can be fixed it will at least minimize the chance of the "browser download confusion" that is the 2nd part of this issue.
- Oshyan
-
This makes me wonder whether it may be some kind of in-between ISP caching. I know someone else mentioned that. Very hard to say indeed.
It's also important to note we may be looking at 2 separate but related issues. On one side certainly there are occasional/intermittent transfer issues, as seen with the PNG issue pointed out earlier in this thread. These would also be likely at the root of, or at least involved in, the reason the installer download problems are occurring. So that issue certainly seems to need investigation and resolution if possible, and it seems that can only be either server-side or at some link in-between since it was reproduced simultaneously on multiple distant systems with (presumably) different Internet providers.
*
*
*
-JavaJones
Was this happening before the servers went over to VMs?
-
Keep up the great ideas, suggestions, clues, etc. Definitely some good ideas on this thread and I know gothic (dc server admin) will be trying some of this stuff to see if it has any effect or triggers any epiphanies.
What are the chances of a file system glitch making the server think the file is smaller than it really is?
Seems unlikely to me -- if that was happening, i would think the server would be experiencing much graver catastrophic crashes in all kinds of operations.-mouser
Not if the corruption is isolated to an area where the OS isn't. Not to mention (if the servers are virtualized) the issue could also exist at the parent machine level.
I had a virtualized mail server that turned into a zero byte file after the power went out once to often. I had recovered it several times before, and typically ran chkdsk on the virtual machine, and then on the parent machine ... But I (got in a hurry) "Forgot" that time and ran it on the parent first. (sh)I.T. happens... :)
-
If it happens again, perhaps folks who can reproduce the problem can capture some of the sessions using something like Wireshark...
-
Was this happening before the servers went over to VMs?
not nearly as often, but yes.. in fact it was. which i think is a real clue.
and it's also what makes me think it's not hardware, since this server is only totally different hardware.
-
This looks like a very promising clue to me, about something that happens when apache web server thread is killed.
http://www.gossamer-threads.com/lists/apache/users/398126
If a server is killed (SIGKILL) during a "large" static file transfer, then the client is not notified by his browser that file has not been completely downloaded. On Win it just says it is not a valid Win32 application..
So my thought is that something might be causing apache processes that are in the middle of serving a file to terminate, causing this problem.
Maybe related: http://osdir.com/ml/web.fastcgi.devel/2003-07/msg00055.html
-
Why are you sending SIGKILL to the server anyway? And if it is some sort of bug, it should probably be a segmentation fault. Either way, I doubt your link to the fastcgi stuff since that message is about 8 years old and I really think that bug has been fixed by now.
-
I don't know much about Apache, but just thinking out loud here. Isn't there a way to tell the server to stop sending checks so that it always forces a download? It isn't an ideal solution, but if it is something like mouser's SIGKILL find, that wouldn't matter if it always proceeded with a new download. Alternatively, couldn't a check file be sent at the end of the downloads and when a new download starts it looks for that file (something like a pad file). If the file doesn't match some criteria (like being present) then it proceeds with a new download. Then a new file signature would need to be added to the end of every new version of a software download, but that is okay...isn't it? Of course I could be talking far too high level here to be useful, but it was an idea as a work around until the culprit could be caught. At very least if it continues you will know it is at the network level and not at the server level.
-
Why are you sending SIGKILL to the server anyway?
sorry if my post was misleading.. what i was suggesting is that it sounded like the bug might be caused when an apache process randomly died or was killed for SOME UNKNOWN REASON.
and then the next logical question would be -- are apache processes being killed or dying in mid-delivery.
but gothic has thrown cold water on this idea, saying that apache processes are not being killed.
-
trying a new setting now.. let's see if it does any good.
-
Why are you sending SIGKILL to the server anyway?-worstje
Might be related to linux OOMw handling?
-
Trying a new setting now -- those of you who were able to recreate the server delivering a partial file error, can you try again and let us know if it's still happening?
-
I downloaded the file without errors, but it also downloaded successfully before the new setting was in place, so it doesn't say much. ;)