DC Server Down for 3 hours on June 4, 2011 due to datacenter power outage

[mouser]

The Seattle Softlayer datacenter where the DC server is located had an unexpected power loss last night (sat 3am-6am) that took the DC server (and DC member server) offline.

Softlayer experienced a power event which caused the Seattle datacenter to go offline. The UPS's failed to pick up the load and engineers are currently investigating the cause. At this time power has been restored, and we are working on bringing all devices back online into working order.
Best Regards, Eric C., SoftLayer Support

The main DC server/forum is now back online (thanks to DC member Gothic!) and all seems well, and no forum data has been lost.


Note 1: The DC member server is still offline for a bit while we attend to that.
Note 2: We' may take the server briefly offline sometime next week to test and improve our daily backup scripts.

[jgpaiva]

I'm now getting this:

An Error Has Occurred!
The server's resources are temporarily under too high a demand to find all the topics you have not read.

Back
DonationCoder.com | About UsDonationCoder.com Forum | Powered by SMF

[ Page time: 14.093s | Server load: 36.66 ]



[edit] not anymore, apparently [/edit]

[mouser]

Yep, give the server a bit of time to finish doing background stuff..

[cranioscopical]

The main DC server/forum is now back online

Thank you Gothic!

[Renegade]

That is bizarre. Don't they have redundant power? Was it a blackout? Brownout?

I host with Softlayer in Texas. They've been excellent.

[Gothi[c]]

That is bizarre. Don't they have redundant power? Was it a blackout? Brownout?

I host with Softlayer in Texas. They've been excellent.

The entire datacenter went dark completely.

Normally they have UPS'es and battery banks to keep things runing until the generators kick in.
The UPS'es "couldn't handle the load" according to them, and everything went down.

It wasn't a brownout as it took quite a bit for them to restore power.

[Ath]

So they are now working on a new disaster recovery plan, I presume

[kyrathaba]

Much appreciation to Gothic, who works tirelessly behind the scenes, keeping DC servers going! 

[mahesh2k]

Is it just me or some good hosts like host gator, soft layer, media temple are facing problems these days ?

[worstje]

I don't know about hosts, but I surely hear a lot of complaints about the place DC is hosted. Professionalism is hard to find, I think, since if they had set it up properly, then they would have had enough juice to last till generators kicked in - or at the very least have had enough juice to power the most important pieces of electronics. Everything going down just means that they messed up big time.

[Renegade]

I've been with The Planet for years, and with SoftLayer and The Planet since they merged.

The number of real problems I have had total exactly zero.

However, I am in the Texas data centers. Maybe that makes a difference.

[worstje]

With data centers, it isn't about how many problems there are over a period. It is about the effect a single problem can have. Good management keeps up on all their expireable UPSes, keeps matters segmented and so forth. That way, if a problem happens, it only hits some customers, or throughput is lowered, and not the entire location. I stick with my opinion that they should have done more disaster tests and drills.

[mouser]

I'm seeing tons of complaints from past SoftLayer fans on their forum and on the webhostingtalk.com forum; seems like a general consensus is forming that since the merger with ThePlanet hosting company, SoftLayer is definitely having some issues..

[Renegade]

I'm seeing tons of complaints from past SoftLayer fans on their forum and on the webhostingtalk.com forum; seems like a general consensus is forming that since the merger with ThePlanet hosting company, SoftLayer is definitely having some issues..

Odd... They were both good companies. Is this just a growing pain do you think?

[mouser]

Is this just a growing pain do you think?

Definitely possible, and it would be understandable.  But it does mean that it might be prudent for people thinking about becoming new SoftLayer customers to wait a bit until things settle down.