topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Thursday March 28, 2024, 4:29 pm
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: August 26 had some slight downtime -- thanks to gothic for fixing it  (Read 21847 times)

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Just a quick note that we had a few hours of downtime this morning, and a big thanks to Gothi[c] for waking up out of his sleep just to fix the problem.  It reminds me how lucky we are to have him taking care of, looking out for, and fixing our server when there are problems.

Thank you Gothi[c] (John).

ps.
I think i will try to set up a very basic backup forum on our dcmembers.com server so we can post when such problems occur and just serve as a place for people to chat if the main server goes down.

Gothi[c]

  • DC Server Admin
  • Charter Honorary Member
  • Joined in 2006
  • ***
  • Posts: 873
    • View Profile
    • linkerror
    • Donate to Member
 :) thanks mousie.

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
I mean it, one of the best things that happend to this site is when you came around and volunteered to take care of these issues.  I know it can be really frustrating work and I want you to know how much we all appreciate it.

TucknDar

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 1,133
    • View Profile
    • Donate to Member
So wasn't just me then :)

Great work, Gothi[c]

Armando

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 2,727
    • View Profile
    • Donate to Member
Thanks for your hard work guys.  :-*

nudone

  • Cody's Creator
  • Columnist
  • Joined in 2005
  • ***
  • Posts: 4,119
    • View Profile
    • Donate to Member
yes, many thanks again to Gothic.

a backup server/forum sounds like a nice idea when troubles happen.

cranioscopical

  • Friend of the Site
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,776
    • View Profile
    • Donate to Member
a big thanks to Gothi[c] for waking up out of his sleep just to fix the problem

That's some dedication there, Gothic, thank you very much!



Shades

  • Member
  • Joined in 2006
  • **
  • Posts: 2,922
    • View Profile
    • Donate to Member
a big thanks to Gothi[c] for waking up out of his sleep just to fix the problem

That's some dedication there, Gothic, thank you very much!

-cranioscopical (August 26, 2008, 04:32 PM)

How so?  I woke up too.... ;)


All jokes aside, cudos to Gothi[c]!!!! :Thmbsup:

jgpaiva

  • Global Moderator
  • Joined in 2006
  • *****
  • Posts: 4,727
    • View Profile
    • Donate to Member
It's good to know that the DC server is on very good hands!
Thank you for your efforts, Gothi[c]!

Also, how do you know the DC server is well treated? The DC server goes down, and mouser doesn't hiperventilate on the irc channel.
The peace of mind warranted by having a great sysadmin is definitelly priceless :D

PhilB66

  • Supporting Member
  • Joined in 2007
  • **
  • Posts: 1,522
    • View Profile
    • Donate to Member
How so?  I woke up too....

All jokes aside, cudos to Gothi[c]!!!! :Thmbsup:

Wake up man, it's kudos  :D

Shades

  • Member
  • Joined in 2006
  • **
  • Posts: 2,922
    • View Profile
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #10 on: August 26, 2008, 09:06 PM »
Sorry, am from the "Give that man a cigar!" generation... ;)

Hmmm, is Cudos not a brand of cigars?  ;D

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #11 on: August 26, 2008, 09:39 PM »
Also, how do you know the DC server is well treated? The DC server goes down, and mouser doesn't hiperventilate on the irc channel.

only the people who were online on irc a couple of years ago when the server had it's first serious outage can know how big a deal this is.  i felt like i was going to have a heart attack i was so panicked and frustrated by the server being down and rushing around trying to figure out what to do.

a little part of my new calmness is just getting used to the fact that servers sometimes have downtime and you don't need to get so worked up, but mostly its the fact that i know John is on the case and will make sure the server is fixed after it has a problem.  it has saved my sanity.

Josh

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Points: 45
  • Posts: 3,411
    • View Profile
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #12 on: August 27, 2008, 01:14 AM »
OK Folks, we now have an official explanation for the problem this morning. If you are a regular of the donationcoder.com IRC channel, you will know that our very own mouser has a problem with air conditioners. Well, today, we have discovered that after a night out on the town, mousey came back to his apartment and mistook his air conditioner for a comode. As such, the air conditioner shorted out. Since mouser runs donationcoder.com off of his 386 running windows nt 3.51 and an aol 2.0 connection, the system overheated due to improper ventilation. This has been resolved, but who knows when we will have another issue with air conditioning. Given mouser's extensive problematic history with air conditioners, this reporter says "Nope".

</comedic satire>

Gothi[c]

  • DC Server Admin
  • Charter Honorary Member
  • Joined in 2006
  • ***
  • Posts: 873
    • View Profile
    • linkerror
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #13 on: August 27, 2008, 07:08 AM »
might still be bumpy today, i'm installing a new kernel today.

cranioscopical

  • Friend of the Site
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,776
    • View Profile
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #14 on: August 27, 2008, 07:37 AM »
the air conditioner shorted out.

What's passed is past, it's water under the southbridge.

Darwin

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 6,984
    • View Profile
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #15 on: August 27, 2008, 08:27 AM »
Many, many thanks, Gothi[c]  :Thmbsup: I was going a bit crazy yesterday morning wondering why DC wouldn't load!

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #16 on: August 27, 2008, 06:05 PM »
So, what caused the glitch, pray tell?
- carpe noctem

Gothi[c]

  • DC Server Admin
  • Charter Honorary Member
  • Joined in 2006
  • ***
  • Posts: 873
    • View Profile
    • linkerror
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #17 on: August 27, 2008, 06:45 PM »
So, what caused the glitch, pray tell?

Ok, here goes a quick one-paragraph un-formatted attempt of explaining:

Well, the conntrack table was getting full, which is normal and common since it's max is rather retardedly low in centos. But increasing the max didn't seem to help. Upon further inspection it seems that some never cleared out of the table. Lowering the conntrack timeout settings didn't help either. I found some references online to an old kernel bug causing connections to not be cleared from there, though I'm not sure if that was the actual cause. It's likely though, since I was heavily monitoring incoming connections and at no point did we get excessively many. So at first to resolve the situation, I figured I'd just unload the ip_conntrack modules and it's depending modules. The only problem is, once loaded, lots of modules depend on it, so I ended up disabling too much of iptable's functionality, causing some weird fail weirdness in the firewall. Then today I actually broke down and recompiled the kernel without connection tracking support, which seems to have fixed it for now. I was hoping for a quick fix by unloading the module, but in the end recompiling the kernel without connection tracking was a quicker fix - typical. Annoyingly, the server went down, like 3 days in a row, every time I go to bed, and sleep for like 1 hour or so. It's like the server waits for the perfect time, stalks me, and then goes down at the worst moment just to bug me :D -- So when I recompiled the kernel, I also just went and grabbed the latest vanilla from kernel.org while I was at it, so to fix any old lurking bugs. So far so good, lets see if it borks out again tomorrow or not :)
« Last Edit: August 27, 2008, 06:46 PM by Gothi[c] »

lanux128

  • Global Moderator
  • Joined in 2005
  • *****
  • Posts: 6,277
    • View Profile
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #18 on: August 27, 2008, 09:24 PM »
just caught this thread. good work, Gothi[c]! i actually didn't notice the downtime, i guess that's advantage of living in a different time-zone. :)

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #19 on: August 28, 2008, 12:22 AM »
Hm, connection tracking... that's a iptables/firewall feature, right? Sounds nasty if such a crucial part of the system is bugged :/
- carpe noctem

Deozaan

  • Charter Member
  • Joined in 2006
  • ***
  • Points: 1
  • Posts: 9,747
    • View Profile
    • Read more about this member.
    • Donate to Member
Re: August 26 had some slight downtime -- thanks to gothic for fixing it
« Reply #20 on: August 31, 2008, 07:00 PM »
Thanks Gothic!