Fixed the MySQL slowness...
This was a really weird one to debug.
When the MySQL sort resorted to a filesort it was actually running FASTER than in memory. (It's supposed to be the other way around).
All MySQL and MyISAM buffers etc were set correctly so it was quite a mystery in which nothing much was making sense.
It was as if the world was turned upside down, and queries that were supposed to be slow in theory, ran faster than those that were supposed to be fast.
It turned out that the problem wasn't with the MySQL configuration at all, but rather an option in the hardened kernel we were running.
( We use a security hardened kernel which prevents many buffer overflow exploits from working through the well known PaX and grsecurity kernel patches )The culprit turned out to be the "prevent invalid userland pointer dereference" PaX option.
When
running a google query for this issue there's only two results, one of which, an
ubuntu hardened guide (pdf), mentions the following:
Restrict mprotect
→ YES if you are running a physical server
→ NO if it is a virtual one
Prevent invalid userland pointer dereference
* If you are running on a physical machine → YES
* If it is a virtual Machine it will slow it down a lot → NO
Glad to see that confirmed. It doesn't really explain the theory as to why unfortunately.
Too bad google only returned this result after we had already found which option caused the problem after lots of debugging
The MySQL query that was taking over 14 seconds to complete now runs in less than a second.
