Cloudflare, a platform that provides network services, was the victim of a DDoS attack last week. It was also accidentally the cause of it.
You might remember Cloudflare was linked to a massive outage in June of this year. When Cloudflare went down, so did sites like Spotify, Google, Snapchat, Discord, Character.ai, and more, all of which rely on Cloudflare’s services. That time, the disruption was sparked by a Google Cloud outage. Earlier this month, Cloudflare had another blunder, albeit much less disruptive than its outage from the summer — but this time, it did it to itself.
“We had an outage in our Tenant Service API which led to a broad outage of many of our APIs and the Cloudflare Dashboard,” Tom Lianza, the vice president of engineering for Cloudflare and Joaquin Madruga, the vice president of engineering for the developer platform at Cloudflare, wrote in a Sept. 13 blog post. “The incident’s impact stemmed from several issues, but the immediate trigger was a bug in the dashboard.”
The bug, according to Lianza and Madruga, caused “repeated, unnecessary calls to the Tenant Service API.” On accident, Cloudflare included a “problematic object in its dependency array” which was recreated, treated as new, caused it to re-run, and, eventually, the “API call executed many times during a single dashboard render instead of just once.”
“When the Tenant Service became overloaded, it had an impact on other APIs and the dashboard because Tenant Service is part of our API request authorization logic. Without Tenant Service, API request authorization can not be evaluated. When authorization evaluation fails, API requests return 5xx status codes,” the blog reads.
Everything is back on track at Cloudflare for now.
“We’re very sorry about the disruption,” the blog post reads. “We will continue to investigate this issue and make improvements to our systems and processes.”
