Fixed the MySQL slowness...

This was a really weird one to debug.

When the MySQL sort resorted to a filesort it was actually running FASTER than in memory. (It's supposed to be the other way around).
All MySQL and MyISAM buffers etc were set correctly so it was quite a mystery in which nothing much was making sense.
It was as if the world was turned upside down, and queries that were supposed to be slow in theory, ran faster than those that were supposed to be fast.

It turned out that the problem wasn't with the MySQL configuration at all, but rather an option in the hardened kernel we were running.
( We use a security hardened kernel which prevents many buffer overflow exploits from working through the well known PaX and grsecurity kernel patches )

The culprit turned out to be the "prevent invalid userland pointer dereference" PaX option.
When running a google query for this issue there's only two results, one of which, an ubuntu hardened guide (pdf), mentions the following:

Restrict mprotect
→ YES if you are running a physical server
→ NO if it is a virtual one

Prevent invalid userland pointer dereference
* If you are running on a physical machine → YES
* If it is a virtual Machine it will slow it down a lot →  NO

Glad to see that confirmed. It doesn't really explain the theory as to why unfortunately.
Too bad google only returned this result after we had already found which option caused the problem after lots of debugging :D

The MySQL query that was taking over 14 seconds to complete now runs in less than a second.

The talk here about sponsored research would imply to a reader that the linked article is sponsored research but I see no indication of that. (unless i missed something?) While I agree that PCWorld is not a good source, and they don't link to the original, it seems to me that there wouldn't be a valid reason to simply dismiss research done by a university as 'not scientific'. (again, unless I'm missing something really obvious, in that case, correct me :D )
Under the comments for the original pcworld article someone linked a bunch of links to other research done on the subject that may be enlightening:

There have been more studies:


http://www.buergerwelle.de/pdf/plantingle.pdf

The Effects Of Microwaves On The Trees And Other Plants

© Alfonso Balmori Martínez. Valladolid. Spain. December, 2003

http://www.emrpolicy.org/science/forum/volkrodt_microwave_smog.pdf

Microwave smog and forest damage; movement in Bonn after all? Volkrodt

http://www.emrpolicy.org/science/forum/volkrodt_elect_pollution_environ.pdf

Electromagnetic pollution of the environment, Volkrodt

http://www.mindfully.org/Technology/Microwaves-Fiasco-Volkrodt1991.htm

http://www.emrpolicy.org/science/forum/volkrodt_fiasco.pdf original document

Are microwaves faced with a fiasco similar to that experienced by nuclear energy? Volkrodt

http://www.emrpolicy.org/science/forum/volkrodt_path_dying_forest.pdf

The Path into the Dying Forest, Ulrich Hertel

http://www.emrpolicy.org/science/forum/volkrodt_letter_yale.pdf

Letter to Professor Wm. Smith, Yale School of Forestry and Environmental Studies, Volkrodt

Enough material, right? :)
I'm not saying wifi kills trees, by the way ;)
PCWorld may well be misquoting the research or jumping to conclusions as it's not clear what type of tests they performed, at what frequencies and what power.
But dismissing it as non-scientific or sponsored when it's not may upset a few researchers that put a lot of work in the study :)

To clarify, our two servers currently have two separate functions (main server and backup server) it's not like one is a backup of the other, though we do have a limited functionality backup forum ready on the member server in case the main server goes down. Our budget is currently not big enough to actually do real HA on two physical servers, or we'd have to get rid of the member server, which wouldn't be very nice either.