Emil Pearce published Novu’s DNS postmortem on 24 March 2025, the day after the cutover. From 22:00 to 22:55 UTC on 23 March, name resolution failed on and off for every user, worldwide. Novu’s product is notifications. For those 55 minutes the names did not resolve.
They were moving authoritative DNS from Amazon Route 53 to Cloudflare. The post does not say how many records moved, which zone, which registrar, or what the old and new TTL numbers were.
The Route 53 → Cloudflare DNS playbook is the zone copy, the delegation, and the records Cloudflare will not store the same way. Novu’s lesson is what still breaks after that copy looks done.
What they did before the window
A week out, they lowered TTLs on the Route 53 records, including the NS record itself. A lower NS TTL is supposed to make resolvers forget the old delegation sooner. They copied the records so the Cloudflare zone matched Route 53. They pointed the registrar at Cloudflare’s nameservers, finished the checks, and flushed caches through Google Public DNS and 1.1.1.1.
They also picked the hour with the least traffic. During the 55 minutes they posted updates on novustatus.com. The postmortem says a review of the procedure will follow. This page is that review, and it does not print a new checklist.
What the resolvers did anyway
Pearce’s root cause is cache timing. Resolvers do not refresh NS records on one clock. Local policy and the TTL they stored both matter, and some of them ignore the TTL you published.
Two things showed up. Some providers kept serving the old delegation. Others briefly served a negative cache. Some ISPs held the old NS records past the TTL Novu had set. Some cached a DNS_PROBE_FINISHED_DOMAIN error, which is the string the post prints. That string is what a browser shows when the lookup has already failed. He is describing the failure users hit, not a DNS response code.
A negative cache is normal DNS. A resolver that gets a temporary miss stores the miss, so it does not hammer the authoritative server. The stored miss lasts for the negative TTL. During this move, a resolver that asked in the gap cached the failure, and those users stayed down until that timer expired. Flushing 8.8.8.8 and 1.1.1.1 clears resolvers Novu can ask. It does not clear the ISP resolver sitting in front of a user. Novu does not operate those.
The intermittence follows from that. One network had already picked up Cloudflare’s nameservers. The next still had Route 53, or a cached miss. The same site was up for one person and down for another inside the same hour.
Why they moved
The reason on the page is performance. Cloudflare’s edge, faster DNS answers, faster SSL termination, and Argo Smart Routing for packet path and congestion. Pearce frames it as one piece of a longer pass on speed, security, and resilience. No before-and-after latency number is in the postmortem, and no follow-up post on Novu’s blog gives one.
Copy the prep list, and add the check the outage shows they still needed. From a network you do not control, resolve the NS set after the registrar change and keep doing it until the old nameservers are gone. A flush of 8.8.8.8 and 1.1.1.1 is not that check. While you watch, treat a cached miss as its own outage. The negative TTL keeps going after the authoritative servers are fine.
Schedule the window when a 55-minute partial outage is survivable. They did, and they still had one. The playbook gets the records across. It does not make every recursive resolver drop a cached miss.