Cloudflare data leak, affecting many sites/services #CloudBleed

[Deozaan]

Cloudflare released an incident report detailing a recent discovery and patching of a bug which leaked data in rare instances. This leaked data includes passwords and other sensitive information.

Virtually every site that uses Cloudflare was possibly affected, meaning that basically you should change your passwords everywhere and make sure you have 2FA enabled where possible. EDIT: See further replies to this thread for clarification on potentially affected sites.

I'm on mobile so it's too much work for me to make things pretty right now, but here are pertinent links:

Cloudflare incident report: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

List of sites (possibly) affected: https://github.com/pirate/sites-using-cloudflare

[wraith808]

That is the view that's going around the web, through various sites and rumours.  What was communicated to customers was different.

Dear Cloudflare Customer:

Thursday afternoon, we published a blog post describing a memory leak caused by a serious bug that impacted Cloudflare's systems. If you haven't yet, I encourage you to read that post on the bug:

https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

While we resolved the bug within hours of it being reported to us, there was an ongoing risk that some of our customers' sensitive information could still be available through third party caches, such as the Google search cache.

Over the last week, we've worked with these caches to discover what customers may have had sensitive information exposed and ensure that the caches are purged. We waited to disclose the bug publicly until after these caches could be cleared in order to mitigate the ability of malicious individuals to exploit any exposed data.

In our review of these third party caches, we discovered data that had been exposed from approximately 150 of Cloudflare's customers across our Free, Pro, Business, and Enterprise plans. We have reached out to these customers directly to provide them with a copy of the data that was exposed, help them understand its impact, and help them mitigate that impact.

Fortunately, your domain is not one of the domains where we have discovered exposed data in any third party caches. The bug has been patched so it is no longer leaking data. However, we continue to work with these caches to review their records and help them purge any exposed data we find. If we discover any data leaked about your domains during this search, we will reach out to you directly and provide you full details of what we have found.

To date, we have yet to find any instance of the bug being exploited, but we recommend if you are concerned that you invalidate and reissue any persistent secrets, such as long lived session identifiers, tokens or keys. Due to the nature of the bug, customer SSL keys were not exposed and do not need to be rotated.

Again, if we discover new information that impacts you, we will reach out to you directly. In the meantime, if you have any questions or concerns, please don’t hesitate to reach out.

Matthew Prince
Cloudflare, Inc.
Co-founder and CEO

Relevant part highlighted. 

Also visit the blog for more information: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/
Initial report: https://bugs.chromium.org/p/project-zero/issues/detail?id=1139

I think that people are trying to make this something that it's not.

We quickly identified the problem and turned off three minor Cloudflare features (email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites) that were all using the same HTML parser chain that was causing the leakage. At that point it was no longer possible for memory to be returned in an HTTP response.

So if you're not using those features, your site was not going through the bad code.

The greatest period of impact was from February 13 and February 18 with around 1 in every 3,300,000 HTTP requests through Cloudflare potentially resulting in memory leakage (that’s about 0.00003% of requests).

[Deozaan]

I think you're right that most likely it's not as bad as it seems. And I admit to reading the "news" about it and posting here before having finished reading the incident report itself, so I missed the part that specifically laid out the requirements for the memory leak. I've adjusted the title of the topic and the content of the original post to be less alarming.

Also visit the blog for more information: https://blog.cloudflare.com/incident-report-on-memory-leak-caused-by-cloudflare-parser-bug/

That's what I linked to in my original post. But thanks for the link to the Google report. I had a hard time finding it myself.

I think that people are trying to make this something that it's not.

We quickly identified the problem and turned off three minor Cloudflare features (email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites) that were all using the same HTML parser chain that was causing the leakage. At that point it was no longer possible for memory to be returned in an HTTP response.

So if you're not using those features, your site was not going through the bad code.

That seems to be the case, but I think it's not entirely accurate:

Because Cloudflare operates a large, shared infrastructure an HTTP request to a Cloudflare web site that was vulnerable to this problem could reveal information about an unrelated other Cloudflare site.

So even if your site doesn't use Cloudflare directly, if it made a request to a site or service that does, then sensitive information from your site could have been leaked.

Also, from the Google report, this is worrisome:

Cloudflare did finally send me a draft [of their incident report]. It contains an excellent postmortem, but severely downplays the risk to customers.

So when I see something like this:

(that’s about 0.00003% of requests)

I have to think that a percentage means nothing to me without knowing how many total requests there were. For an exaggerated example, 0.00003% of 38 quintillion is still quite a lot.