the crazy thing is we are paying for an expensive hosting company (softlayer) specifically because they are supposed to be one of the most reliable companies with the best redundancies, etc.
Well then, I would recommend (if you don't mind, and I'm not trying to teach you to suck eggs), from l-o-n-g experience of IT service contacts on both sides (customer/supplier), that you carefully scrutinise your contract and/or SLA (Service Level Agreement) for conditions and in particular penalty clauses
in the event of deteriorated service or loss of service.
If there are
any penalty clause provisions, then I think (from memory) you could legally claim either of:(a) actual
or reasonable notional consequential costs, or loss of revenue/profit arising from the outage.
damages ("And don't do it again!" type damages)
- but not both.
If there isn't
any penalty clause, then you may have unwittingly signed a contract with no teeth for the customer in the event of an outage such as this. A contract for "All care and no responsibility".
If you get no financial recompense for the outage, and if you believe that you are:
... paying for an expensive hosting company (softlayer) specifically because they are supposed to be one of the most reliable companies with the best redundancies, etc.
- then I'd suggest that you may have been paying out money under false pretences for as long as you have been using that supplier, and should swap suppliers because of that fact alone, and ask for a full/partial refund.
If you told your account manager/rep. that you were considering this, then it might be interesting to see what sort of response that gets.
- A favourable (to you) response would probably indicate that they are interested in holding onto your business.
- An unfavourable response would probably give the lie to any notions or expectations of "customer care and QOS" that you might have held regarding this supplier.
As to the outage itself, if it really shouldn't have happened
because the supplier - to your knowledge - had all the appropriate redundancies/backups in place, then - by definition - that could means that there was a process failure somewhere.
In my experience the main thing that usually gets in the way of a good service manager providing his services to meet an SLA is a pencil-head (usually an accountant). So be on the lookout for that as a possibility. They may have been cost-cutting and hoping to get away with it.