DonationCoder.com New Server Migration Coming Soon

[mouser]

Ok we are going to be moving the DC domain to a new server in the next couple of days (over the weekend probably).

This could go completely smoothly, or may hit some hiccups, and may mean a day or two of strangeness as the new server address propagates through the domain name servers.

I will post again right before we pull the trigger, and then the forum will close while we migrate the database, and then re-open on the new server.

So everyone grab your teddy bears and baby cody plushies and hold tight over the weekend, and wish Gothi[c] good luck as he drives the truck full of dynamite over the bridge.

SERVER CHANGEOVER HAPPENING TODAY - SUNDAY 12/12/10

We did not go ahead with the server changeover due to difficulties getting the mysql database operations optimized.  See post below.

[KynloStephen66515]

Mouser should have added:

If you have any problems getting to the forums over the weekend, please visit the IRC chat room

[mouser]

I should note that we aren't planning any site/forum changes at this time.. so the migration should be fairly painless and you shouldn't see any real differences.  [some more substantial site changes are planned for 2011 though!]

[y0himba]

I should note that we aren't planning any site/forum changes at this time.. so the migration should be fairly painless and you shouldn't see any real differences.

 
Famous last words anyone? 

[some more substantial site changes are planned for 2011 though!]

As my teenager would say: "OMGOMGOMGOMGOMGOMGOMG!" I can't wait to see what's coming!

[rgdot]

Wasn't there an alternate site set up for when main DC goes down? I have it bookmarked but it's lost in the maze that is my bookmarks.

[app103]

Also, you can follow DC on twitter, friendfeed, and facebook. If anything goes wrong, those would be good places to find out news.

[mouser]

SERVER CHANGEOVER HAPPENING TODAY - SUNDAY 12/12/10

[Carol Haynes]

Good luck (or has it already happened - in which case well done!)

[mouser]

Visit the temporary standalone chat page to join the chat channel (irc #donationcoder on efnet) if the forum is closed while we are migrating.

[f0dder]

Drumroll!

[40hz]

And because it will run as a VM, it will cease to exist in any real sense and become an imaginary place.

Going forward, every night at midnight, it will also turn into a pumpkin. "For maintenance."

Ah virtual servers! "Nothing is real. And nothing to get hung about."

[Stoic Joker]

Ah virtual servers! "Nothing is real. And nothing to get hung about."

A Beatles quote? The last thing we need now is buggs ... You'll get us in a Strawberry (fields) jam.

[cranioscopical]

it will run as a VM, it will cease to exist in any real sense

You're  a real nowhere man!

[40hz]

it will run as a VM, it will cease to exist in any real sense

You're  a real nowhere man!

Actually, what I really want is to be a paperback writer!

Seriously.

[cranioscopical]

Actually, what I really want is to be a paperback writer!

Seriously.

Well you know we all wanna change the world!


  good for you!

[mouser]

This is still the old server..
We had everything reproduced and tested on the new server, EXCEPT we could not get the new mysql 5.1 database stuff running at a good speed.  Problem is being worked on and we will move when it is solved.  Obviously the last thing we want is to move servers and have the forum feel slower.

Apologies for the downtime!

We are back up on the same server as we have been -- the move to the new server will happen later this week probably.
Thank you for your patience.

[techidave]

It's too bad you had some problems with the switch over.  But I am glad that you didn't keep it there while trying to work the problems out.

We have faith in you guys that you will soon get the problem resolved! 

[cranioscopical]

We had everything reproduced and tested on the new server, EXCEPT we could not get the new mysql 5.1 database stuff running at a good speed.  Problem is being worked on and we will move when it is solved.  Obviously the last thing we want is to move servers and have the forum feel slower.

Thanks for the update, and thanks for planning things so that this kind of glitch can be dealt with elegantly!

The gem cannot be polished without friction, nor man perfected without trials.

[f0dder]

EXCEPT we could not get the new mysql 5.1 database stuff running at a good speed.

Are you switching from an older version to 5.1 and getting bad speed, or is it 5.1-on-old-hardware to 5.1-virtualized that's giving you trouble?

[mouser]

We are moving from mysql5.0x to mysql5.1x, and moving from a native 4gb linux install (centos) to a virtual machine (VmWare ESX) gentoo install on a machine with 8gb.  Both using same Dual Processor Quad Core cpu.

We seem to have tracked down the problem to mysql performance when sorting larger tables..
We've played with dozens of mysql settings with very little effect.

More details as we figure out more.

[40hz]

FWIW the forums seem to be loading noticeably quicker today than previously. At first I chalked it up to the transition until I read your notice. Did you do some heavy duty clean-up or optimizations to the drives or database prior to trying the transition?

I checked the stats on my network and router and they're in line with their averages so I suspect the improvement I'm seeing is due to something you've done on your end.

Luck!

[Carol Haynes]

What a bummer - good luck later on in the week! 

[Gothi[c]]

Fixed the MySQL slowness...

This was a really weird one to debug.

When the MySQL sort resorted to a filesort it was actually running FASTER than in memory. (It's supposed to be the other way around).
All MySQL and MyISAM buffers etc were set correctly so it was quite a mystery in which nothing much was making sense.
It was as if the world was turned upside down, and queries that were supposed to be slow in theory, ran faster than those that were supposed to be fast.

It turned out that the problem wasn't with the MySQL configuration at all, but rather an option in the hardened kernel we were running.
( We use a security hardened kernel which prevents many buffer overflow exploits from working through the well known PaX and grsecurity kernel patches )

The culprit turned out to be the "prevent invalid userland pointer dereference" PaX option.
When running a google query for this issue there's only two results, one of which, an ubuntu hardened guide (pdf), mentions the following:

Restrict mprotect
→ YES if you are running a physical server
→ NO if it is a virtual one

Prevent invalid userland pointer dereference
* If you are running on a physical machine → YES
* If it is a virtual Machine it will slow it down a lot →  NO

Glad to see that confirmed. It doesn't really explain the theory as to why unfortunately.
Too bad google only returned this result after we had already found which option caused the problem after lots of debugging

The MySQL query that was taking over 14 seconds to complete now runs in less than a second.

[cranioscopical]

Well done, we're lucky to have you!

[mouser]

we're lucky to have you!

amen -- gothi[c] has spent days working on this.. when he gets his teeth into an issue, he just does not let go until he solves it.  we are *VERY* lucky to have him as part of DC and taking such good care of our servers.