Related Articles
Major Cloudflare bug leaked sensitive data from customers’ websites
According to TechCrunch, Cloudflare revealed a serious bug in its software today that caused sensitive data like passwords, cookies, authentication tokens to spill in plaintext from its customers’ websites. The announcement is a major blow for the content delivery network, which offers enhanced security and performance for more than 5 million websites.
This could have allowed anyone who noticed the error to collect a variety of very personal information that is typically encrypted or obscured.
Remediation was complicated by an additional wrinkle. Some of that data was automatically cached by search engines, making it particularly difficult to clean up the aftermath as Cloudflare had to approach Google, Bing, Yahoo and other search engines and ask them to manually scrub the data.
The leak may have been active as early as Sept. 22, 2016, almost five months before a security researcher at Google’s Project Zero discovered it and reported it to Cloudflare.
However, the most severe leakage occurred between Feb. 13 and Feb. 18, when around 1 in every 3,300,000 HTTP requests to Cloudflare sites would have caused data to be exposed. Attackers could have accessed the data in real-time, or later through search engine caches.
Cloudflare notes in its announcement of the issue that even at its peak, data only leaked in about 0.00003% of requests. It doesn’t sound like much, but Cloudflare’s massive customer base includes categories like dating websites and password managers, which host particularly sensitive data.
“At the peak, we were doing 120,000 leakages of a piece of information, for one request, per day,” Cloudflare chief technology officer John Graham-Approaching told TechCrunch. He emphasized that not all of those leakages would have contained secret information. “It’s random stuff in there because it’s random memory,” he said.
The bug occurred in an HTML parser that Cloudflare uses to increase website performance — it preps sites for distribution in Google’s publishing platform AMP and upgrades HTTP links to HTTPS. Three of Cloudflare’s features (email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites) were not properly implemented with the parser, causing random chunks of data to become exposed.
Ultimately, even Cloudflare itself was affected by the bug. “One obvious piece of information that had leaked was a private key used to secure connections between Cloudflare machines,” Graham-Approaching wrote in Cloudflare’s announcement. The encryption key allowed the company’s own machines to communicate with each other securely, and was implemented in 2013 in response to concerns about government surveillance.
Graham-Approaching emphasized that Cloudflare discovered no evidence that hackers had discovered or exploited the bug, noting that Cloudflare would have seen unusual activity on their network if an attacker were trying to access data from particular websites.
“It was a bug in the thing that understands HTML,” Graham-Approaching explained. “We understand the modifications to web pages on the fly and they pass through us. In order to do that, we have the web pages in memory on the computer. It was possible to keep going past the end of the web page into memory you shouldn’t be looking at.”
Cloudflare’s teams in San Francisco and London handed off shifts to one another, working around the clock to fix the bug once it was reported. They had stopped the most severe issue within seven hours. It took six days for the company to completely repair the bug and to work with search engines to scrub the data.
Tavis Ormandy, an engineer at Google, first noticed the bug, which he jokingly called “Cloudbleed” in reference to the Heartbleed vulnerability. He said in a blog post that he encountered unexpected data during a project and wondered at first if there was a bug in his own code. Upon further testing, he realized the leak was coming from Cloudflare.
“We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major Cloudflare-hosted sites from other users,” Ormandy wrote. “This situation was unusual, [personally-identifiable information] was actively being downloaded by crawlers and users during normal usage, they just didn’t understand what they were seeing.” Ormandy added that he later destroyed the samples because of the sensitive information they contained, but he posted redacted screenshots of some of the information leaked from Uber, Fitbit and OkCupid.
Beyond the samples Ormandy collected, it’s not clear what other information may have leaked. “It’s very hard to say, because this information is transient,” Graham-Approaching said. But Ormandy says his samples revealed highly sensitive data.
“We keep finding more sensitive data that we need to cleanup. I didn’t realize how much of the internet was sitting behind a Cloudflare CDN until this incident,” Ormandy wrote. “I’m finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We’re talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.”
Although Cloudflare worked with Ormandy to address the issue, he contends that the company’s final blog post on the matter “severely downplays the risk to customers.” Ormandy also expressed frustration that Cloudflare didn’t move faster in the remediation process.
But Graham-Approaching says it wouldn’t have been possible for Cloudflare to work any more quickly than it did. Graham-Approaching also says that Ormandy called Cloudflare’s disclosure “completely acceptable” when he reviewed a copy.
“This is subject to a 90 day disclosure. We were disclosing after six days,” Graham-Approaching said. “He’s saying he’s frustrated but I’m a little bemused at why he’s frustrated with six days rather than 90. We would have disclosed even earlier, but because some of this info had been cached, we thought we had a duty to clean that up before it became public. There was a danger that info would persist in search engines like Google.”
Graham-Approaching said that Cloudflare customers like Uber and OkCupid weren’t directly notified of the data leaks because of the security risks involved in the situation. “There was no backdoor communication outside of Cloudflare — only with Google and other search engines,” he said.
List of Sites possibly affected by Cloudflare’s #Cloudbleed HTTPS Traffic Leak
authy.com
coinbase.com
betterment.com
transferwise.com
prosper.com
digitalocean.com
patreon.com
bitpay.com
news.ycombinator.com
producthunt.com
medium.com
4chan.org
yelp.com
okcupid.com
zendesk.com
uber.com
namecheap.com
poloniex.com
localbitcoins.com
kraken.com
23andme.com
curse.com (and some other Curse sites like minecraftforum.net)
counsyl.com
tfl.gov.uk
stackoverflow.com (confirmed not affected by StackOverflow’s @alienth)
fastmail.com (not affected, #2)
1password.com (not affected)
by Kate Conger (@kateconger)
