Avaiability/outages of the DCF website.

[IainB]

Is there a log of service level incidents/outages of the website?
I got an Error 522 today, reported by Cloudflare  - copied per the attached file which is just text in an .mhtml file, in a .zip file.
I wondered whether it was a known outage/incident or an unknown intermittent error of some kind.
Thought I should report it.
 There was a diagram that showed the connections between:
  Browser (me) <---> Cloudflare (Tokyo) <---> Host (donationcoder.com)

 - with the Browser and Cloudflare shown as "working" and their link OK, but the link between Cloudflare to Host was X'd out (not working).

I did a Ctrl-R (refresh) and after a rather longish wait, the DCF site came up OK.

[Shades]

After the DDOS attack not too long ago, I understood that the DC website now uses CloudFlare's facilities to handle internet traffic to the DC web server.

CloudFlare indicates that error 522 can be caused by:
 -   Overloaded web server
 -   Offline origin web server
 -   Blocked Cloudflare requests
 -   Faulty network routing
 -   Disabled keepalives
 -   Incorrect IP address in the Cloudflare DNS settings (i.e. the request from us was sent to the wrong place)
 -   Dropped packets on the host network

The responsibility for the first 2 possible causes are directly related to DC, the other possible causes are more on CloudFlare's turf. Therefore CloudFlare can be just as much a solution as it can be a problem.

[KynloStephen66515]

Generally, when the site goes down like that, it's cause mouser has pressed the wrong button on something

Sometimes though it can just be a small hiccough at the datacenter, causing the network to lose connection for a few moments.

[IainB]

Well, I wasn't as concerned with having an analysis of that specific incident and its causes/responsibilities per se as much as I was with simply identifying the correct reporting path  - e.g., does DCF maintain an Incident Log (per ITIL good/best practice)? - but I have no real idea what the process for reporting such incidents for DCF might be in any case.

Is there a log of service level incidents/outages of the website?

-IainB (December 22, 2018, 11:24 AM)

Anyway, I've reported it now.   

[rgdot]

Generally, when the site goes down like that, it's cause mouser has pressed the wrong button on something

-Stephen66515 (December 22, 2018, 04:25 PM)

[IainB]

@rgdot: Yes, the Captain might typically press the wrong button, as @Stephen66515 suggests:

Generally, when the site goes down like that, it's cause mouser has pressed the wrong button on something

-Stephen66515 (December 22, 2018, 04:25 PM)

Reminds me of an auditor I knew who worked at a banking data processing organisation. He was being proudly shown around the operations-room of one of the several new distributed national data centres when he suddenly and inadvertently became notorious for being "that guy who curiously pressed an unlabelled big red button on the side of an IBM mainframe box". I'm not sure what the purpose of the big red button was, or why it was prominent in an area where people could touch it or knock against it if it was such a risk, but his pressing it apparently resulted in the shutdown of the whole data centre for a couple of hours.

He kept his job because the button was unlabelled (came with no warnings), was easily accessible in a "safe" (read, "safe for monkeys to roam") area in the first place, and his action had clearly highlighted a serious potential operations-room process risk - which was subsequently rectified.
Seemed fair to me. (True story.)