Report on server downtime on August 8th

[mouser]

The server was the subject of some kind of mail attack yesterday, August 8th. and was down for most of the day.

It looks like our current host (we pay $210 a month for this account!! yipes!) is not staffed by people who have a good grasp on things (www.servstra.com) as i've written about before and i don't think they know what happened or how to prevent it from happening again, so it may be that we will get attacked in the exact same way in the coming days.

Basically it looks like some spammer used random fake email addresses from donationcoder.com to spam people, and so all the messages were bouncing back to us.  It's not clear yet to me (nor i guess to the server admins) whether this spammer somehow actually got our server to send these emails out to random addresses, or whether they just managed to spoof our address on the return address.  I will report more as I find out more.

If so, don't panic!  I am always on top of it and doing my best to get us up asap, and the site will never be down for long.  what doesn't kill us just makes us stronger.  and in the world of hosting, what does kill us only kills us for a little while.

Remember in the early days of the site a year ago we had some growing pains as we moved to a dedicated server, but once we got it sorted we had a blazing fast server and user space.  Well it seems now we are experiencing a bit more growing pains and I'm working to split us up onto 2 servers, a main server for the website, and an auxiallary server for user accounts, which we can have fun on and give our members some space for their own stuff.

Anyway, back to work and fun.  And thank you for being understanding.  The only thing I can tell you is that I try my heart out to get us up as fast as possibile and I'm always working to try to make this site and server better, faster, stronger, more stable, etc., but it's a learning experience for me.  I appreciate the understanding and patience.

[AbteriX]

Thank you for all of your support  Very appreciated 
It was a dull morning without DC.

what doesn't kill us just makes us stronger

-Mr. Mouser

The same proverb we have in germany too

[Carol Haynes]

I guessed it might be a DoS attack or something similar.

I have been getting increasing amounts of crap to my domain names (random user names) and lots of 'undeliverable' receipts from spoofed email addresses based on my domain names.

I think I will point my sites back to my domain name manager and then use web forwarding to my domain space on the DC servers. That way I can manage mail via their servers and it will reduce the amount a crap (at least a little) that is filtering through the DC servers. It souldn't make much difference to me in the long run (just accessing my domains will need an extra hop or two).

Glad it is all sorted for the mo!

[mouser]

i am going to hire us a professional server management contract so we will get to a state where we are well cared for and protected by people who know what they are doing - i have no intention of resting until our servers are as stable as is humanly possible.

[nudone]

great to see everything back to normal - DC does leave a very noticeable hole when it's not there.

will a new host be sought or are there advantages staying put?

[mouser]

a new host is being sought, no doubt about it.
ive already got a new secondary server set up and im migrating all secondary accounts as we speak (donationcoders.com etc.)

after i finish that and learn how to do it smoothly without major interruption i will move to stage two, getting us a new primary server, and hiring a proper management company to handle it as if it was filled with state secrets.

[nudone]

fantastic. 

[mouser]

i think the best way to view these outages are as little lessons.  lessons that can teach us how to improve stability and security, and ensure that we have skilled people who know how to handle issues.  we will be the better for it in the long term when it is important.

i'm actually extremely excited about splitting our stuff onto 2 servers, one primary one for the main site and another one dedicated to hosting user accounts and stuff.  it will make it 100x more secure for us, and make providing web space/blogging accounts, etc, to users much easier and worry free.  it's a bit more money on server costs of course, but well worth it for peace of mind and for the flexibility we will have in providing services for members.

[nudone]

i agree, these are very valuable lessons - better to learn (or begin learning) now rather than at a later time when DC is perhaps even more of a target for attack - drawing more attention will no doubt bring more nuisance in at the same time.

all in all, it looks like a bright DC future. i'm sure everyone is relieved you are on top of it all, mouser.

[housetier]

* housetier gives mouser some love to put his heart at ease 

[tsaint]

simple one:thanks VERY much mouser for all your time and worry and effort you're putting into this.
On the bright side I managed to do a bit extra work at school today because of the down time. I think at least one of my classes benefitted.
tony

[mouser]

i appreciate very much the supportive words, and again i do apologize to everyone who has a site on donationcoder.com server.  what i can tell you is this, if it's any consolation: the moment there is any trouble on the server, i am planted in front of the screen and contacting tech support and talking to people constantly, as long as it takes, to get the problem fixed. i take nothing for granted and assume that such problems require my constant attention until they get fixed.  i know it doesnt make up for the site being offline but i just want you guys to know that i do not take such things lightly and i am sitting in front of the screen for the entire day trying to get it solved as quickly as possible.

[nudone]

cody would just like to express how he feels about the current DC hosts and the recurring server downtime.

[AbteriX]

@nudone
  

[housetier]

great one nudone  

[mouser]

nudone that is priceless..  as i said earlier, we have to get all these cody illustrations into a flash game.

[nudone]

(note that cody has taken off the red stockings he was so fond of wearing.)

here's a quick sketch of cody's first reaction to the DC server downtime - be warned. do not view if you are easily upset.

Spoiler

[f0dder]

Good job, mouser.

I hate those spam/flood/etc kiddies >_<

[Josh]

Mouser, if you have a chance, hop on irc. I would like to help you examine any logs you have available to further determine the exact happenings since your hosting company is of no use.

Good to see us back online thou! I was lost without dc