News and Reviews > Official Announcements
August 26 had some slight downtime -- thanks to gothic for fixing it
Darwin:
Many, many thanks, Gothi[c] :Thmbsup: I was going a bit crazy yesterday morning wondering why DC wouldn't load!
f0dder:
So, what caused the glitch, pray tell?
Gothi[c]:
So, what caused the glitch, pray tell?
--- End quote ---
Ok, here goes a quick one-paragraph un-formatted attempt of explaining:
Well, the conntrack table was getting full, which is normal and common since it's max is rather retardedly low in centos. But increasing the max didn't seem to help. Upon further inspection it seems that some never cleared out of the table. Lowering the conntrack timeout settings didn't help either. I found some references online to an old kernel bug causing connections to not be cleared from there, though I'm not sure if that was the actual cause. It's likely though, since I was heavily monitoring incoming connections and at no point did we get excessively many. So at first to resolve the situation, I figured I'd just unload the ip_conntrack modules and it's depending modules. The only problem is, once loaded, lots of modules depend on it, so I ended up disabling too much of iptable's functionality, causing some weird fail weirdness in the firewall. Then today I actually broke down and recompiled the kernel without connection tracking support, which seems to have fixed it for now. I was hoping for a quick fix by unloading the module, but in the end recompiling the kernel without connection tracking was a quicker fix - typical. Annoyingly, the server went down, like 3 days in a row, every time I go to bed, and sleep for like 1 hour or so. It's like the server waits for the perfect time, stalks me, and then goes down at the worst moment just to bug me :D -- So when I recompiled the kernel, I also just went and grabbed the latest vanilla from kernel.org while I was at it, so to fix any old lurking bugs. So far so good, lets see if it borks out again tomorrow or not :)
lanux128:
just caught this thread. good work, Gothi[c]! i actually didn't notice the downtime, i guess that's advantage of living in a different time-zone. :)
f0dder:
Hm, connection tracking... that's a iptables/firewall feature, right? Sounds nasty if such a crucial part of the system is bugged :/
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version