Author Topic: Preparing for a new master DonationCoder server (Read 33,078 times)

mouser · « **on:** November 20, 2010, 01:11 AM »

DC server admin Gothi[c] and I have been discussing the possibility of moving to a new server for quite some time, and it looks like we are getting ready to pull the trigger and do it.
We still have more research to do and more decisions to make, so we thought we would start a thread where people could chime in.

Let me tell you the starting point for how we are looking at this.

First, our current setup is as follows:
We have a $279/month dedicated main server, hosted by SoftLayer; they are very reliable and very good, but on the high price side of things. It's a Dual Processor Quad Core Xeon 5430.
We have a $100/month dedicated member server, hosted by netdepot; we've had some service issues with them but they have been really good about price and let you buy down up front costs to reduce monthly costs.

Having two servers has been important in letting us keep the main server fast and secure, and putting less secure and lower priority stuff on the member server.

But Gothi[c] has been really pleased with the new VMWare operating system setups that can be done on servers now, which allow you to run multiple virtual servers with their own operating systems, completely and securely isolated from one another.

And we have some other things we'd like to do with a new server setup, to improve security/scalability, and to make new projects easier in the future (such as creating a new virtual machine for secure hosting of experimental projects, etc.).

SO.. our current thinking is to transition from these two servers, to a new single server, running multiple virtual machine operating systems, and the thing we are debating now is the best way to organize the one master server into multiple virtual machines..

Gothi[c] · « **Reply #1 on:** November 20, 2010, 01:24 AM »

So, there's a few ways we can go about this....

One suggestion:

* main dc server http on a vm (minus mail)
* dcwing member server on a vm (minus mail)
* vm for main dc email
* vm for member server email
* vm for newsletter sending
* (optional - vm for sql )
* vm for commercial project hosting

The main reasoning behind putting the newsletter on it's own vm (and more importantly IP) is so that should we ever unjustly be considered spammers due to the high volume mail of the newsletter and end up on some blacklist, the mail forum registration mails won't stop working (ie end up in people's junk mailboxes). This vm wouldn't have to run all the time, only during newsletter sending time.

The reasoning for seperating SQL on a separate server is security and scalability. A compromised sql server won't touch the main server's files. (Same reason why email would be separate), it also allows us to tune the OS setup towards the specific task. (eg, mail would have a larger /var partition for the mail queue, and perhaps a filesystem that performs better with lots of smaller files)

Currently we already have MySQL and Apache isolated on the member server using BSD jails. It works but I feel there is some performance hit due to the need for NAT in pf. Having separate vm's would be a much cleaner solution.

There's many other ways we could go at this...
The mail vm's could all be diskless OS'es with a master server so that 1 OS upgrade migrates to all the vm's automatically for example. (It would also save a lot of drive space). But then the problem of course is that if the master server goes down, all the nodes will too. (We could put them on a NAS if we can afford it, though that's up in the air right now)

If any one has good ideas on what services to separate in which vm's lets hear them. It's more fun if we can involve everyone in building our new home

Gothi[c] · « **Reply #2 on:** November 20, 2010, 02:03 AM »

So, post more graphs like this:

[attachment 0 not found]

worstje · « **Reply #3 on:** November 20, 2010, 02:07 AM »

It's always good to see movement in the serverpark to know stuff is not left to bitrot simply because it works.

However, I feel it warrants saying - you put a lot of focus on the 'what if something breaks' department. Whereas before you had a pair of servers on two different locations (if I understand it properly), now you intend to scale it all into a single physical machine. Suppose some mechanical doodah goes down the crapper, how easily can you move one of those VMs to another physical machine? (That is supposed to be one of the strengths in a VM, after all, right?)

Shades · « **Reply #4 on:** November 20, 2010, 04:23 AM »

@ worstje:
Movability will be dictated by the backup plan that is in use or will be used. Besides that, there could be a redundant server there that is ready to take over.

Now I do agree with worstje that DC is putting all off Cody's eggs in one basket with this scenario.

If it was me I would consider getting two PC's with slightly lighter hardware spec (which normally drops the price of them quite a bit) and divide the load. In case one of them craps out there is always the other PC that can take over, while the 'crapper' can be repaired/replaced.

From the graphic I understand that the load could easily be 'smeared' over two PC's. Now it is not known to me if DC has their own dedicated server parked with either ISP or that they could buy the one that they are using now. Anyway, maybe it is an idea to use that one as the backup and buy the new PC as specified by Gothi[c].

That way everybody can be happy and confident that the essence of DC website always will be served to the community (as long as the electrical grid supplies the power of course).

mouser · « **Reply #5 on:** November 20, 2010, 04:37 AM »

One of the original motivations for having two servers was so that if one crashed, we could run a temporary forum on the other server.

Theoretically that makes a lot of sense, but in practice that doesn't seem like it's that much of a concern.. we could run a backup forum anywhere on any free forum hosting site should that happen, and the more important concern is having a good backup/restore system in place, and be hosted someplace with extremely good uptime and reliability.

Gothi[c] · « **Reply #6 on:** November 20, 2010, 06:22 AM »

To clarify, our two servers currently have two separate functions (main server and backup server) it's not like one is a backup of the other, though we do have a limited functionality backup forum ready on the member server in case the main server goes down. Our budget is currently not big enough to actually do real HA on two physical servers, or we'd have to get rid of the member server, which wouldn't be very nice either.

mouser · « **Reply #7 on:** November 30, 2010, 05:35 PM »

Just a quick update.. we actually got the new server!
Gothic is setting it up as we speak.

We have not switched over to it yet.. we plan to do so sometime around the end of dec or january.

The new server should have a number of benefits, including making it easier for us to keep on top of security updates, making it easier to maintain rigorous backups, and giving us the security and freedom to set up some experimental new web services.

Thank you to all donators whose donations have made it possible for us to upgrade the server.

(When all is said and done our monthly server costs shouldn't be much more than they are now, as we are basically consolidating servers).

40hz · « **Reply #8 on:** November 30, 2010, 06:39 PM »

Out of curiosity, which VMWare product are you using for this? Are you just running server or did you spring for something bigger like Sphere?

mouser · « **Reply #9 on:** November 30, 2010, 06:51 PM »

I believe it's vmware ESX.. but gothic can fill us in on all the details.

Gothi[c] · « **Reply #10 on:** December 03, 2010, 01:04 PM »

vSphere is pretty expensive.
We are using ESXi, which is free.

Rover · « **Reply #11 on:** December 03, 2010, 04:23 PM »

The nice thing about VM's is they are so portable, replaceable, etc.

If you run on VMware primarily, you can keep an EC2 image for backup.

1) Set up all of your VM's so everything is working nice
2) Save copies of the VMs as EC2 VM's
3) Do regular data backups and keep it safe
4) update your EC2 copies as you make significant change to your VM

DR Solution:
1) Create new EC2 accounts as needed.
2) Put your Back EC2 VMs on the account
3) Restore data
4) Begin taking backups from EC2 systems

You should then have a work solution with relatively recent data.

Recovery:
1) Spin up your VM's (You backed them up, right?)
2) Restore data
3) Stop EC2 Accounts
4) Start going backups again

Just my

Migrating from VMware to EC2

steeladept · « **Reply #12 on:** December 03, 2010, 05:17 PM »

Rover beat me to it, but I would look at more than EC2. Since you already are planning on ESXi as your basis, there are A LOT of cloud providers that can provide temporary DR/HA capabilities for a price. I don't know details (never needed to research it before) but I know that is one of the selling points for the cloud. And in almost every case, they do this by leveraging VMware. There are a few exceptions - Microsoft's Hyper-V is used by a few vendors, for example, but by and large you can copy and drop your entire environment to their service to accomplish these capabilities at a fraction of the cost. I have even heard of companies who complete a real world, proven, DR plan in minutes because they have an off-site SAN replicated in real-time (or near real-time) that provides the VM data to remote machines. Granted they still pay a lot, for VMware ESX if nothing else, but it is possible.

I am just saying that if you are looking at DR (backup) and HA-like abilities, and since you are already talking ESXi as your platform, consider looking into a hosted service option as your temporary site if/when you need to implement these features.

mouser · « **Reply #13 on:** December 05, 2010, 07:19 PM »

Gothic has been hard at work configuring the new server.. and that's exciting even though it may not appear very different once we switch over to it.

But it seems to me that it might be time for a re-imagining of the DonationCoder.com website.

I'm not talking about the forum itself, i'm talking about the rest of the static pages on the site, which are currently static html.

I'm thinking we might move to a proper CMS.

Should we have a little redesign competition to see what ideas people can come up with for the new look?

Ashraf · « **Reply #14 on:** December 06, 2010, 07:21 AM »

Mouser,

Out of curiosity what are the specs on the new sever, aside from the CPU? How much bandwidth are you getting from SL?

In regards to CMS, I would say go Wordpress (even though it isn't officially a CMS) but Wordpress doesn't integrate that well with Simple Machines AFAIK.

mouser · « **Reply #15 on:** December 06, 2010, 02:16 PM »

Specs:
Intel Xeon Quad Core Dual Processor (Xeon-Harpertown 5430-Quadcore 2.66GHz) [same as before]
8gb ram [used to have 4gb]
dual sata 500gb drives in raid configuration [used to have dual 250gb not in raid]
1000mbs uplink [same as before]
3000gb bandwidth
Vmware ESX Operating System [used to be CentOs]

Note that while we used to have separate main server and much smaller member server, now we are moving everything to one server and using virtual machines.

40hz · « **Reply #16 on:** December 06, 2010, 03:51 PM »

This is more under site design than graphic design:

I would very much like to see some sort of revival of the weekly (Dr. Ehtyar/Stephen) "Tech Newz" feature - except I think it would be better if it were implemented as a blog rather than a forum post this time around.

A rolling format would be more practical since it's difficult for readers to comment on stories under the "single topic per" paradigm forum threads use. Note: I'm only talking about the tech news section here. The forum itself is fine. It's the weekly newspost that doesn't work well inside the forum.

Any individual story/discussion that really took off could always be moved over to a forum thread to keep major discussions all in one place and avoid creating a split betwen the forum and the newsdesk areas. You could always do a link to the news blog from within the forum to help provide a more unified user experience.

A blog engine would also make it easier for the news editor to post stories as they break - or as the news editor's time permits - rather than on some arbitrary weekly deadline. We don't "go to print" so there's really no need to have press deadlines or "editions" anymore.

Just thinking out loud.

Actually, a blog format would make a lot of sense for the annual NANY as well as the software author's pages. Once you load up something like Wordpress, you can easily create as many blog spaces as you'd need and still have the regular forum and static pages for what they work better for. But this is a topic for a whole 'nuther discussion...and... uh...I'm gonna shut up now.

mouser · « **Reply #17 on:** December 06, 2010, 04:27 PM »

My great fear with using a separate blog system (as we did early on in DC existence when we had regular "columnists") is that the different sources of content (forum posts, blogs) are pretty walled-off from one another and it creates a very weird schizophrenic situation where it's hard to know what should go where and who is seeing what.

Don't forget that we already have a homebrew "blog" system in place for Tech News and other "featured" items.. it's basically a simple system i created to let me elevate forum posts to blog entries in different categories.

It doesn't have many of the bells and whistle's of a proper blog software, but it has one *huge* advantage for us, in that it avoids duplicate content and helps to unify the blog and forum content. This is really one of the great unresolved issues of internet content management in my view (and we have discussed this before), namely the issue of exposing content in different ways for different purposes and users.

As an example of the homebrew blog interface to featured forum posts here see:

Main DC Blog: http://blog.donationcoder.com
TechNews: http://www.donationcoder.com/technews/
Coding Snacks: http://www.donationcoder.com/CodingSnacks

superboyac · « **Reply #18 on:** December 06, 2010, 04:30 PM »

I agree with 40hz. I also think some of the people here can have their own blogs here that would be useful. I think the idea to go from featured blog to discussion forum in some kind of unified manner would be really cool.

Anyway, this is very interesting to me. So it costs roughly $400/month to run the dc servers. Interesting. Nothing good ever comes free, eh?

superboyac · « **Reply #19 on:** December 06, 2010, 04:32 PM »

well...mouser is right also. I don't know...I shouldn't really participate in this, this is not my bag

.

mouser · « **Reply #20 on:** December 06, 2010, 04:42 PM »

Quote

So it costs roughly $400/month to run the dc servers

Yep. actually about $500 a month when you include server administration costs.. which let's us give gothic (DC server admin) some money for taking such good care of the servers, installing stuff, setting up new servers and user accounts, updating software and watching for security issues, and most importantly be available when something goes really wrong.

The important thing to say is that it's only because people continue to support the site with their donations that we are able to afford these servers, which keep the site fast and allow us to provide web space for member coders, etc. Some of you may remember the days in 2005 when we were on a $20/month shared hosting server and things could get SLLOOOWWWW... Thankfully those days are long gone.

So once again thank you to all who have donated and whose donations make the site possible

superboyac · « **Reply #21 on:** December 06, 2010, 04:45 PM »

It's amazing that this runs on donations. I just keep thinking about people making so much money for doing so much less. Ahhh....

mouser · « **Reply #22 on:** December 06, 2010, 05:18 PM »

Well we are going to have to have another fundraiser very soon, so everyone set aside a few bucks for DC please!

Now back to our regularly scheduled discussion..

rgdot · « **Reply #23 on:** December 06, 2010, 05:47 PM »

Keep up the great work, a bit that I can afford will be sent soon-ish (don't ask what soon-ish means

)

mouser · « **Reply #24 on:** December 06, 2010, 05:55 PM »

Rover, can you explain a bit more about EC2 -- i couldn't really figure out what it was about from searching the web -- specifically how it relates to vmware.