NanoNeuron
Worked example ·

Two Cloudflare outages, in postmortem form

The template is easier to judge with a real incident in it. These two are public, unusually well documented by the company that ran them, and checkable by anyone with the links.

What this is, and isn't

This page was assembled by hand from Cloudflare's two published incident reports. Nothing here is drafted by our tool, and nothing is paraphrased: each field of the template is filled only with a sentence quoted verbatim from the report and linked back to it. Where the report doesn't state something the template asks for, the field says so instead of filling the gap.

It is not an independent analysis of Cloudflare, and it doesn't imply Cloudflare uses this product. It is their account, in this shape. For what the tool itself produces, unedited, see the GitHub outage demo.

18 November 2025 -- core traffic failed globally

Source: Cloudflare outage on November 18, 2025, by Matthew Prince, on the Cloudflare blog. Every quoted line below is from that post.

Severity

The report doesn't assign one. On our own scale this is a sev1: a global failure of core traffic delivery lasting hours.

Impact

On 18 November 2025 at 11:20 UTC (all times in this blog are UTC), Cloudflare's network began experiencing significant failures to deliver core network traffic. This showed up to Internet users trying to access our customers' sites as an error page indicating a failure within Cloudflare's network.

Timeline (UTC), as published

TimeStatusDescription
11:05Normal.Database access control change deployed.
11:28Impact starts.Deployment reaches customer environments, first errors observed on customer HTTP traffic.
11:32-13:05The team investigated elevated traffic levels and errors to Workers KV service.The initial symptom appeared to be degraded Workers KV response rate causing downstream impact on other Cloudflare services. Mitigations such as traffic manipulation and account limiting were attempted to bring the Workers KV service back to normal operating levels. The first automated test detected the issue at 11:31 and manual investigation started at 11:32. The incident call was created at 11:35.
13:05Workers KV and Cloudflare Access bypass implemented -- impact reduced.During investigation, we used internal system bypasses for Workers KV and Cloudflare Access so they fell back to a prior version of our core proxy. Although the issue was also present in prior versions of our proxy, the impact was smaller as described below.
13:37Work focused on rollback of the Bot Management configuration file to a last-known-good version.We were confident that the Bot Management configuration file was the trigger for the incident. Teams worked on ways to repair the service in multiple workstreams, with the fastest workstream a restore of a previous version of the file.
14:24Stopped creation and propagation of new Bot Management configuration files.We identified that the Bot Management module was the source of the 500 errors and that this was caused by a bad configuration file. We stopped automatic deployment of new Bot Management configuration files.
14:24Test of new file complete.We observed successful recovery using the old version of the configuration file and then focused on accelerating the fix globally.
14:30Main impact resolved. Downstream impacted services started observing reduced errors.A correct Bot Management configuration file was deployed globally and most services started operating correctly.
17:06All services resolved. Impact ends.All downstream services restarted and all operations fully restored.

Detection

The first automated test detected the issue at 11:31 and manual investigation started at 11:32. The incident call was created at 11:35.

Root cause

A change in our underlying ClickHouse query behaviour (explained below) that generates this file caused it to have a large number of duplicate 'feature' rows.

Contributing factors

When the bad file with more than 200 features was propagated to our servers, this limit was hit -- resulting in the system panicking.

Resolution

We solved the problem by stopping the generation and propagation of the larger-than-expected feature file and replace it with an earlier version of the file.

Action items

Not reproduced here. Cloudflare's report lists its own follow-up work; it belongs to them and is best read at the source.

5 December 2025 -- 28% of HTTP traffic, 25 minutes

Source: Cloudflare outage on December 5, 2025, by Dane Knecht, on the Cloudflare blog. Every quoted line below is from that post.

Severity

Not assigned in the report. On our scale a sev2: a large share of traffic, but contained within half an hour by a revert.

Impact

On December 5, 2025, at 08:47 UTC (all times in this blog are UTC), a portion of Cloudflare's network began experiencing significant failures. The incident was resolved at 09:12 (~25 minutes total impact), when all services were fully restored.

A subset of customers were impacted, accounting for approximately 28% of all HTTP traffic served by Cloudflare

Timeline (UTC), as published

TimeStatusDescription
08:47INCIDENT startConfiguration change deployed and propagated to the network
08:48Full impactChange fully propagated
08:50INCIDENT declaredAutomated alerts
09:11Change revertedConfiguration change reverted and propagation start
09:12INCIDENT endRevert fully propagated, all traffic restored

Detection

08:50 -- INCIDENT declared -- Automated alerts

Root cause

it was triggered by changes being made to our body parsing logic while attempting to detect and mitigate an industry-wide vulnerability

Contributing factors

When the killswitch was applied, the code correctly skipped the evaluation of the execute action, and didn't evaluate the sub-ruleset pointed to by it.

Resolution

The issue was identified shortly after the change was applied, and was reverted at 09:12, after which all traffic was served correctly.

Action items

Not reproduced here. Cloudflare's report lists its own follow-up work; it belongs to them and is best read at the source.

What the filled-in shape makes obvious

Both reports put the detection gap in plain numbers: 11:28 impact, 11:31 first automated detection in November; 08:47 deploy, 08:50 declared in December. Both name the thing that stopped the impact (a restored file; a revert) separately from the thing that caused it. And in November, the first ninety minutes of investigation went to Workers KV, which the report itself describes as the initial symptom rather than the trigger -- the timeline records that plainly, which is exactly what a timeline is for.

That's the discipline the template asks of your own incidents: every claim points at a row, and a row nobody recorded at the time is a gap you write down rather than a story you reconstruct. If you'd rather the citing were done for you, your first incident is free.

Back to PostMortem AI · The template · How it works