So, what caused the glitch, pray tell?
Ok, here goes a quick one-paragraph un-formatted attempt of explaining:
Well, the conntrack table was getting full, which is normal and common since it's max is rather retardedly low in centos. But increasing the max didn't seem to help. Upon further inspection it seems that some never cleared out of the table. Lowering the conntrack timeout settings didn't help either. I found some references online to an old kernel bug causing connections to not be cleared from there, though I'm not sure if that was the actual cause. It's likely though, since I was heavily monitoring incoming connections and at no point did we get excessively many. So at first to resolve the situation, I figured I'd just unload the ip_conntrack modules and it's depending modules. The only problem is, once loaded, lots of modules depend on it, so I ended up disabling too much of iptable's functionality, causing some weird fail weirdness in the firewall. Then today I actually broke down and recompiled the kernel without connection tracking support, which seems to have fixed it for now. I was hoping for a quick fix by unloading the module, but in the end recompiling the kernel without connection tracking was a quicker fix - typical. Annoyingly, the server went down, like 3 days in a row, every time I go to bed, and sleep for like 1 hour or so. It's like the server waits for the perfect time, stalks me, and then goes down at the worst moment just to bug me
-- So when I recompiled the kernel, I also just went and grabbed the latest vanilla from kernel.org while I was at it, so to fix any old lurking bugs. So far so good, lets see if it borks out again tomorrow or not
