We use Cloudflare to serve ~20-30TB of traffic a month where I work. Was the SRE on call when I got paged on our blackbox monitoring/third party web checks failing..
It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet.
I wish more companies would take note of the transparency and sense of urgency on updating their status page. (Looking at you Azure)
Your experience is different to mine. Here in Australia the status page was inaccessible. Which immediately leads to common wisdom on monitoring systems - the monitor must always be separate to the monitored.
Are you using Cloudflare DNS? I couldn't access it either during the outage and I just came to the realisation it might be because I switched to 1.1.1.1 a while ago.
Cloudflare's status page is one of the very few services of Cloudflare that are hosted elsewhere precisely for the reason it needs to be available if something isn't working.
It seemed like ISPs with east coast Vocus backhaul (going to SYD rather than MEL/PER POPs - TIL there's also CBR/BNE/ADL) had it worst trying to get to the status page but as the timeline shows, cloudflare was probably already recovering as we were triaging.
The default way that most networking devices are managed is crazy in this day and age.
Like the post-mortem says, they will put mitigations in place, but this is something every network admin has to implement bespoke after learning the hard way that the default management approach is dangerous.
I’ve personally watched admins make routing changes where any error would cut them off from the device they are managing and prevent them from rolling it back — pretty much what happened here.
What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection.
Many devices still rely on “not saving” the configuration, with a power cycle as the rollback to the previous saved state. This is a great way to turn a small outage into a big one.
This style of device management may have been okay for small office routers where you can just walk into the “server closet” to flip the switch. It was okay in the era when device firmware was measured in kilobytes and boot times in single digit seconds.
Globally distributed backbone routers are an entirely different scenario but the manufacturers use the same outdated management concepts!
(I have seen some small improvements in this space, such as devices now keeping a history of config files by default instead of a single current-state file only.)
> What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection.
Always good. The system built into display settings ("click yes if you can read this, or the change will be reverted in 15 seconds") has saved me a number of times. No reason not to apply that to other settings where the data channel is the same as the control channel.
Luckily a lot of modern networking equipment has an automated rollback feature you can key off of. For instance junos based devices have a commit confirmed <time> where it will auto rollback if not confirmed after X amount of time. Still pretty dated but they are designed to be reliable first.
> What should be the default on every networking device is a two-stage commit where the second stage requires a new TCP connection.
OpenWRT router distribution has had this for years, it's amazing! (As is OpenWRT)
OpenWRT also has SQM CAKE which saved my sanity on parents DSL connection for years. As far as congestion control and bandwidth sharing goes, nothing else compares
The power cycle as a rollback is IMO reasonable. If you're talking about equipment in a data center you should presumably have some sort of remote power management on a separate network.
Alternatively some sort of watchdog timer would be a great addition (e.g. rollback within X minutes if the changes are not confirmed).
Yes/no – something like a watchdog timer should be part of whatever OS the router is running. Remote power management would be closer to bespoke but ought not to be too exotic.
Even 20 years ago at a nationwide ISP we used to have a completely separate emergency management network over GSM. I'm amazed that at CF's scale management is still done the way described.
If what's presented in the blog is actual configuration that's very much a junos device and definitely has the ability to have commit with confirm, auto rollback, and commit history. Not using it is more of an issue with the automation, as they said.
(Which is an issue with the /automation/ defaults. I've learned enough to do commit confirm, but by default ansible does a hard commit.)
One of our sites uses Cloudflare and serves 400k pageviews per month and generates around $650/day in ad and affiliate revenue. If the site is not up the business is not making any money.
Looking at the hourly chart in Google Analytics (compared to the previous day) there isn't even a blip during this outage.
So for all the advantages we get from Cloudflare (caching, WAF, security [our WP admin is secured with Cloudflare Teams], redirects, page rules, etc) I'll take these minor outages that make HN go apeshit.
Of course it helped that most our traffic is from the US and this happened when it did but in the past week alone we served over 180 countries which Cloudflare helps make sure is nice and fast :D
I didn't quite understand this. It sounds like Cloudflare's outage didn't affect you depite being their customer. Why did their large outage not affect you?
We still get several hundred pageviews per hour during the outage. It just didn't seem to effect us much for some reason (but the reason is not that nobody was going to our site anyways)
I did get an alert from Uptime Robot but when I checked everything was fine and so I thought it was a false positive.
I am in Lisbon and was not having trouble because Cloudflare's Lisbon data center was not affected. But over in Madrid there was trouble. It depended where you are.
From the article: "Depending on your location in the world you may have been unable to access websites and services that rely on Cloudflare. In other locations, Cloudflare continued to operate normally."
I did not notice Cloudflare going down. Only reason I knew was because of this thread. Either it was because I was asleep, or my local PoP wasn't affected.
Not OP, but my team really, really enjoys using a combination of Segment.io for event tracking and piping that data into Amplitude for data viz, funnel metrics, A/B tests, conversion, etc.
07:42: The last of the reverts has been completed. This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically.
Now I'm remembering the story of how, when a certain blue website fell off the Internet for a day ~a decade and a half ago (due to some slightly broken database migration logic), out-of-band access boiled down to who was still logged in (!): https://rachelbythebay.com/w/2019/01/20/quiet/
Amusingly when things went wrong again last year, it was BGP's fault (is this the hyperscale equivalent of "it's always DNS" or something?). Engineers (with adequate credentials) had to actually drive to the datacenter haha.
I would be very interested to hear more about how the break-glass process worked.
This was something I was surprised not to see directly addressed in terms of follow up steps. When discussing process changes, they mention additional testing, but nothing to address what seems to be a significant communication gap.
I'm sure they have a more detailed internal postmortem, and I imagine it'd go into that. This is a nice high-level overview. They probably don't want to bury that under details of their communication processes, much less go into exactly who did what when for wide consumption by an audience that may not be on board with blameless postmortem culture.
You're probably right, but if they're going to mention it as part of the problem, I would want to see it as part of the solution. However, I agree that they certainly shouldn't name names.
I think I experienced first-hand the moment those network engineers were reverting their own reverts, breaking the web again. For example, DoorDash.com had come back online, then went back to serving only HTTP 500 errors from Cloudflare, then came back online again. I raised it in the HN discussion and @jgrahamc responded minutes later.
I'd be super interested in understanding what this means concretely. For example, are we talking about reverting commits? If so, why were engineers reverting reverts?
also can happen when your deploy process has two flows for revert a forward movement revert (where new bits and head are committed fixing the items that needed to be reverted) and a "previous head" revert which just goes back one revision in the rcs (or tagged version).
Imagine the first eng team did a forward movement revert that corrected the issue and had a new head bits that gets deployed, where shortly after another eng fires off the second process type and tells the system to pull back to the last revision (which is now the bad revision as it was just replaced with fresher deploy bits).
Having two revert processes in the toolkit and maybe a few disperse teams working to revert the issue without tight communication leads to this issue.
I think this is more likely the basis issue vs a bad merge (I assume that the root cause was broadcasted wide and large to anyone making a merge)
In a world where it can take weeks for other companies to publish a postmortem after an outage (if they ever do), I never ceases to amaze me how quickly CF manage to get something like this out.
I think it's a testament to their Ops/Incident response teams and internal processes, it builds confidence in their ability to respond quickly when something does go wrong. Incredible work!
To further add to your point, the CTO is the one who shared it here & the CEO is incredibly active on forums & social media everywhere with customers. Communication has always been one of their strengths.
I do wonder what would happen should happen if either of them left the company, I feel like there's a lot of trust on HN (and other places) that's heavily attached to them as individuals and their track record of good communication.
Good communicators generally foster that environment. And their customers appreciate it, so there is an external expectation now too. Everything ends some day, but I think this will be regarded as a valuable attribute for awhile.
Devil's advocate, you could get taken over or end up with a different board. I wouldn't like to see it but someone's got to compete with you or we'll have to send in the FTC! :)
No provider is perfect, but it's because of stuff like this that I trust Cloudflare waaaaaaaaaaay more than the likes of Amazon. Transparency engenders trust, and eventually, love! Thank you, Cloudflare.
The sheer level of technical competence of your engineering team continues to astound me. (Yes, they made a mistake and didn't catch an error in the diff. But your response process went exactly as it should, and your postmortem is excellent.) I couldn't even begin to think about designing or implementing something of this complexity, much less being able to explain it to a layperson after a failure. It is really impressive, and I hope you will continue to do so into the future!
Most of the companies I've worked for unfortunately don't use your services, but I've always been a staunch advocate and converted a few. Maybe the higher-ups only see downtime and name recognition (i.e. you're not Amazon), but for what it's worth, us devs down the ladder definitely notice your transparency and communications, and it means the world. I've learned to structure my own postmortems after yours, and it's really aided in internal communications.
Thank you again. I can't wait for the day I get to work in a fully-Cloudflare stack :)
AWS is pretty decent if you're in an NDA contract (you have paid support). You can request RCAs for any incident you were impacted and they'll usually get them within a day.
Not as transparent as "post it on the internet" but at least better than the usual hand wavey bullshit
I feel like others lose opportunities by not doing the same. By publishing early and publishing the details they: keep the company in the news with positive stuff (free ad), get an internal documentation of the incident (ignoring the customer oriented "we're sorry" part), effectively get a free recruitment post (you're reading this because you're in tech and we do cool stuff, wink), release some internal architecture info that people will reference in discussions later. At a certain size it feels stupid not to post them publicly. I wonder how much those posts are calculated and how much organic/culture related.
I agree that this is a free ad/recruitment. However, it’s easy to see how more conservative businesses see this as a risk. They are highlighting their deficiencies, letting their big important clients know that human error can bring their network down.
Additionally, these post-mittens work for Cloudflare because they have a great reputation and good uptime. If this were happening daily or weekly, it would be a warning sign to customers.
It’s a strategy other companies could adopt, but to do it effectively requires changes all across the organization.
OTOH, I think most actual engineers would know that everywhere has deficiencies and can be brought down by human error, and I'd personally rather use a product where the people running it admit this rather than just claim that their genius engineers made it 100% foolproof and nothing could ever possibly go wrong
1. On the buy side, crappy big companies with procurement etc. may not have “actual engineers” deciding things. Maybe for something like Cloudflare that’s likely to sit within an actual technical team’s mandate
2. Not 100% foolproof but if a startup is selling its tool and has an uptime page and details of all its downtime and as a prospective customer you go and see that they have 3-4 hour downtimes once or twice a month, it should raise alarm bells.
>I feel like others lose opportunities by not doing the same
IMO it is a slippery slope to see this as opportunity too strongly. Sure, doing the right thing may be net beneficial to the business in the long run...but the $RIGHT_THING should be done first and foremost because it's the right thing.
Ehhhh… I think it’s good (for us) that they do this, but I don’t think it’s a free ad (contrary to popular belief, not all news is good news, and this is bad news) and any sort of conversion rate on recruitment is probably vanishingly small (which would normally be fine, but incidents like these may turn off some actual customers, which is where actual revenue comes from).
I think their calculation (to the extent you can call it that) is that in the interest of PR and damage control, it’s better to get a thorough postmortem out quickly to stem the bleeding and keep people like us from going “I can’t wait to hear what happened at Cloudflare” for a week. Now we know, the customers have an explanation, and this bad news cycle has a higher chance of ending quickly.
> but incidents like these may turn off some actual customers
Incidents - yes. But why would a post-mortem turn someone off? The incident happened regardless. Do you think anyone would be more likely turned off by reading how they solved it / plan to prevent it on the future than by silence?
> I wonder how much those posts are calculated and how much organic/culture related.
Don't companies have a fiduciary duty to calculate things; the reason for doing something actually cannot just be that it's a nice thing to do? Not down to the word, but at least the general decision to be this way?
No they don't have such duty. In practice very little decision making is based on hard data in my experience. Real world being fuzzy and risk being hard to quantify do not help the situation.
I agree, I think the transparency builds trust and I encourage it where I can. The counter thought I had when reading this case though, is it almost feels too fast. What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done.
Doing incident response (both outage and security), the tactical fixes for a specific problem are usually pretty easy. We can fix a bug, or change this specific plan to avoid the problem. The search for conditions that allowed the incident to occur can be alot more time consuming, and most organizations I've worked for are happy to make a couple tactical changes and move on.
What I mean by that is I hope there isn't an incentive to wrap up the internal investigation quickly and write the blog and send it, and go we're done.
There is not. From here there's an ongoing process with a formal post-mortem, all sorts of tickets tracking work to prevent further reoccurrence. This post is just the beginning internally.
I have to agree. The environment that leads to a fast blog post may also lead to this quote from the post:
> This was delayed as network engineers walked over each other's changes, reverting the previous reverts, causing the problem to re-appear sporadically.
They are running as fast as they can and this extended the incident. There is a “slow is smooth, smooth is fast” lesson in here. I’d rather have a team that takes a day to put up the blog post, but doesn’t unnecessarily extend downtime because they are sprinting.
There's normal operating procedure and sign offs and automation etc. etc. and then there's "we've lost contact with these data centers and normal procedures don't work we need to break glass and use the secondary channels". In that situation you are in an emergency without normal visibility.
It can be easy to arm-chair it afterwards, but unless things can be done in parallel (and systems should be designed so this can be done, things like "we're not sure what's wrong, we're bringing up a new cluster on the last known good version even as we try to repair this one") you have to make a choice, and sometimes it won't be optimal.
At a previous job, I worked with two guys who were excellent in a crisis. One of them used to run operations, the other was a crusty old programmer who’d been around for a while. I tried to learn as much as I could from both of them.
At around the same time, I was watching HBO’s The Wire. One scene had a high-profile shooting with a frantic police response; people were running everywhere trying to help. The lead commander on the scene gave this instruction to his sergeant: “Slow this thing down to a crawl. Give these bastards no chance to fuck up in a meaningful way.”
And then it hit me. That’s exactly how they ran the calls. I asked them about this, and they said absolutely – human nature, when things are broken badly, is biased towards action. You want to try to make a change, to reboot a system, to do something to hopefully make things better. But often if you were not careful, you risk losing information about the outage you are in. Best case scenario you luck into fixing the problem and don’t know how. Worst case? You’ve changed the state of an already broken system, and have done nothing but add more variables to unwind.
So now, every time I’m on an outage call, I try to do what Wes and Tim and Major Rawls would all do: I take control, pump the brakes, and make sure that we are capturing enough information about the current state that we don’t confuse ourselves further.
>take weeks for other companies to publish a postmortem
And with nowhere near the detail level of what was presented here. Typically lots of sweeping generalizations that don't tell you much about what happened, or give you any confidence they really know what happened or have the right fix in place.
Well cloudflare’s entire value is in uptime and preventing outages. Showing they have a rapid response and strong fundamental technical understanding is much more critical in the “prevent downtime” business.
To be fair though they sort of MUST do things like this to have our confidence - their whole business is about being FAST and AVAILABLE. Were not talking about Oracle here :-D
There was a common pattern in use back in the day when I managed openbsd filewalls (can't remember if it was ipf or pf days). When changing firewall rules over ssh, you'd use a command line like:
$ apply new rules; sleep 10; apply original rules
If your ssh access was still working and various sites were still up during that 10sec you were probably good to go - or at least you hadn't shut yourself out.
Back when I was a briefly a network engineer at the start of my career, on cisco equipment we'd do 'reload in 5' before big changes - so it'd auto restart after 5 minutes unless cancelled.
I'm sure there were and are better ways of doing it, but it was simple enough and worked for us.
most ISP tier routers have an entire commit engine to load and apply configs.
junipers allows for instance, one to do the command commit confirmed, which will apply the configuration, and revert back to the previous version if one does not acknowledge this command within a predifined time.
this prevents permanent lockout out of a system.
Yet another BGP caused outage. At some point we should collect all of them:
- Cloudflare 2022 (this one)
- Facebook 2021: https://news.ycombinator.com/item?id=28752131 - this one probably had the single biggest impact, since engineers got locked out of their systems, which made the fixing part look like a sci-fi movie
"Due to this withdrawal, Cloudflare engineers experienced added difficulty in reaching the affected locations to revert the problematic change. We have backup procedures for handling such an event and used them to take control of the affected locations."
But Cloudflare had sufficient backup connectivity to fix it. I'm curious how Cloudflare does that today-- the solution long ago was always a modem on an auxiliary port.
Not sure how common it is, but you can get serial OOBM devices accessible over cellular which would then give you access to your equipment.
I'm surprised more places don't implement a "click here to confirm changes or it'll be rolled back in 5 minutes" like all those monitor settings dialogues
You say that like it hasn't been going on since the mid 1990's, when it got deployed.
I'm not blaming BGP, since it prevents far more outages than it causes, but BGP-based outages have been a thing since its beginning. And any other protocol would have outages too - BGP just happens to be the protocol being used.
These are the public facing BGP announcements that cause problems, but doesn't account for the ones on private LANs that also happen. Previous employers of mine have had significant internal network issues because internal BGP between sites started causing problems. I'm not sure there's anything better (I am not a network guy), but this list can't be exhaustive.
There are lots of other causes of incidents, like cut cables, failed router hardware, data centers losing power etc.
It just seems that most of these are local enough and the Internet resilient enough that they don't cause global issues. Maybe the exception would be AWS us-east-1 outages :-)
Maybe a testament to BGP's effectiveness that so many large-scale outages are due to misconfiguring BGP rather than the frequent cable cuts and hardware failures that BGP routes around.
Except that this wasn't an example of BGP being prone to error or fragile. This was, as the blog post specifically calls out, human error. They put two BGP announcement rules after the "deny everything not previously allowed" rule. It's the same as if someone did this to a set of ACLs on a firewall.
The main difference between BGP and all other tools is that if you mess up BGP, you've done a very visible thing because BGP underpins how we get to each other's networks. But it's not a sign of BGP being fragile, just very important.
That does seem like bad UX/"DevX" that that configuration of rules is "valid" syntactically and there weren't better equivalents of "linters"/"compilers" flagging that before it ever got sent out as an announcement. UX issues are a "proneness" to error/fragility. It sounds like there is room to build a "higher level language" (like a "Typescript : Javascript :: ? : BGP") for BGP announcements that is less prone to "accidentally bad programs". Not that I have immediate suggestions, just that my gut reaction from skimming these sorts of outage reports is that if it was a "language" I was writing in I can hear that I'd want a lot more (type) safety nets.
Some tools are more prone to human error than others.
Another canonical example is C++. Some tools make it easy to blow your leg off. Some tools provide safety mechanisms to stop the saw from cutting off your finger.
Time and time again, this type of response proves that it's the right way handle a bad situation. Be humble, apologize, own your mistake, and give a transparent snapshot into what went wrong and how you're going to learn from the mistake.
Or you could go the opposite direction and risk turning something like this into a PR death spiral.
Exactly. I trust businesses/people that are transparent about their mistakes/failures much more than the ones that avoid them (except Apple which never accepts their mistakes, but I still trust their products, I think I'm affected by RDF).
At the end of the day, everybody makes mistakes and that's okay. Everybody else also know that everybody makes mistakes. So why not accept it?
I really don't get what's wrong with accepting mistakes, learning from them, and moving on.
> I really don't get what's wrong with accepting mistakes, learning from them, and moving on.
Some people really struggle with this (myself included) but I think it's one of the easiest "power ups" you can use in business and in life. The key is that you have to actually follow through on the "learning from them" clause.
I agree, but lately (as in the past month) I've been finding myself using apple maps more and more than google. When on a complicated highway interchange, the 3d view that Apple Maps gives for which exit to take is a life-saver
Recently I used Apple Maps much more than Google Maps.
In addition to trying to de-Googlify my life, there was also an occurance where Google Maps literally tried to kill me: at an intersection that connects into a highway it guided me to drive straight into the opposite direction to a highway, straight onto the coming cars at 140km/h. I've quit Google Maps right there and never used it again.
I would agree with that. Apple Maps was worse than the hockey puck mouse or the trashcan macpro. trying to decide if it is worse than the butterfly keyboard, but I think the keyboard wins for the shear fact that it impacted me in a way that was uncorrectable where I could just use a different Maps app
They didn’t apologize about the direction the pro macs were going a few years back but they certainly listened and made amends for it with the recent Pro line and MacBook Pro enhancements
This is a great concise explanation. Thank you for providing it so quickly
If you forgive my prying, was this an implementation issue with the maintenance plan (operator or tooling error), a fundamental issue with the soundness of the plan as it stood, or an unexpected outcome from how the validated and prepared changes interacted with the system?
I imagine that an outage of this scope wasn’t foreseen in the development of the maintenance & rollback plan of the work.
I lead the platform team of a fairly young startup in the D2C commerce space in the APAC region. This outage happened during peak traffic hours which made me and the team look like amateurs in the company.
Cloudflare is great, and I would never move away from it. But from a business continuity standpoint, is there a fallback approach that we should be prepared for during such cases?
One crude approach we were discussing is during an outage we could change the NS records in the registrar to point to for eg. Google Cloud DNS which would already be in sync in terms of the DNS records it has.
If you're okay with load balancing DNS queries across multiple providers you could do 2x Cloudflare primary NS, 2x GCP for example where each provider is in sync with each other.
If not, a manual swap at the registrar level would be good enough.
I should also mention this approach sort of breaks with Cloudflare's proxied records which dynamically assign anycast IPs for records placed on their CDN. So if using this approach the failover NS provider would probably need to also use a different CDN, preferably one that just gives you a CNAME.
Cedexis now part of Citrix offers a multi-cdn product, that allows you to load balance between CDNs, check them out. I'm not sure who else is in that space, but in general it sounds like you want a multi-cdn strategy.
Every outage represents an opportunity to demonstrate resilience and ingenuity. Outages are guaranteed to happen. Might as well make the most of it to reveal something cool about their infrastructure.
Where does one even start with learning BGP? It always seemed super interesting to me, but not really something that could be dealt with on a small scale, lab type basis. Or am I wrong there?
You can simulate arbitrarily large networks and internetworks with this, provided you have the hardware to run a large enough number of virtual appliances, but they are pretty lightweight.
Mininet is what the Georgia Tech OMSCS Computer Networking labs use. It's not bad, the two labs that stood out to me were using it to implement BGP and a Distance Vector Routing protocol.
DN42 <https://dn42.eu/Home> gets mentioned a lot. Its basically a big dynamic VPN that you can do BGP stuff with. Pretty cool but I could never get my node working properly.
CF is the only company I have ever seen that can have an outage and get pages of praise for it. I don't have any (current) use for CloudFlare's products but I would love to see the culture that makes them praiseworthy spread to other companies.
I think a lot of companies don't realize the whole "Acknowledging our problems in public" thing CF got going for it is a positive. Lots of companies don't want to publish public post-mortems as they think it'll make them look weak rather than showing that they care about transparency in the face of failures/downtimes.
As others have said, this is a clear and concise write up of the incident. That is underlined even more when you take into account how quickly they published this. I have seen some companies take weeks or even months to publish an analysis that is half as good as this.
Not trying to take the light away from the outage, the outage was bad. But the relative quickness to recovery is pretty impressive, in my opinion. Sounds like they could have recovered even quicker if not for a bit of toe stepping that happened.
I think it's even better that they explained the backgorund of the outage in a really easy to understand way, so that not only experts can get a hang of what was happening.
I was on a severely understaffed edge team fronting several thousand engineers at a fortune 500 - every deploy felt like a spacex launch from my cubicle. I have a lot of reverence for the engineers who take on that kind of responsibility.
I read the blog twice and have some thoughts:
The root cause seems is as: "While deploying a change to our prefix advertisement policies, a re-ordering of terms caused us to withdraw a critical subset of prefixes."
And a dry-run: "a Change Request ticket was created, which includes a dry-run of the change, as well as a stepped rollout procedure."
And a Peer review: "Before it was allowed to go out, it was also peer reviewed by multiple engineers. "
I would doubt the expertise of tech guys of cloudflare, reviewing the change. And there was a dry-run.
But is it really OK to apply the change to a spine network which would affect 50% network traffic? Just out of peer review and a dry run? No green/blue, no gray release, maybe these are not proper for a small change here. But this "small" change really got big affect. I thougt it was worth it.
And from my shallow experience, the dry-run would always have do nothing to the env. It is dry-run anyway.
And at last the three lines are found out. So I wonder how did this re-order happen? And why?
With these tiny changes, there should be some mechanism to verify their correctness, not just review and dry-run.
We use a phased rollout process for all routine changes (like this one). Once a change has passed peer review and the "dry-run", changes are rolled out to progressively larger slices of our production environment, with monitoring systems and engineers watching for adverse effects.
The specific network locations that were impacted by this change were amongst the last to see the change rolled out. One deficiency in our deployment strategy (which we will correct) is that no network locations in the affected "MCP" configuration received the change early in our rollout process. If that had been the case, we would have found the problem much earlier and the incident's impact would have been much reduced.
Most of the criticisms seem to be around BGP and network management. What I’m seeing here that also is important is that the change was applied to a DC where the route change didn’t trigger the defect. In essence, this is also due to a very classic problem of a test dataset giving a false sense of security due to variation from other configurations. For this reason my team prefers to rollout changes to production using a test region that most customers don’t use yet will have some visible impact if there’s any error in our presumptions so far such as hard-coding regions and relying upon services not present or as capable across all regions. This practice has caught a number of rather serious errors for us that while customer impacting was nowhere near as bad as if we had rolled out simply randomly like many teams do essentially. This is even more important the more difficult it is to perform rollbacks of changes or for rollbacks to take effect such as DNS and CDN caching changes.
Would be great if the timeline covered 19 minutes of 6:32 – 06:51. How long did it take to get the right people on the call? How long did it take to identify deployment as a suspect?
Another massive gap is the rollback: 6:58 – 7:42 – 44 minutes! What exactly was going on and why did it take so long? What were those back-up procedures mentioned briefly? Why engineers where stepping on each other toes? What's the story with reverting reverts?
Adding more automation, tests and fixing that specific ordering issue of course is an improvement. But that adds more complexity and any automation ultimately will fail some day.
Technical details are all appreciated. But it is going to be something else next time. Would be great to learn more about human interactions. That's where the resilience of a socio-technical system happened and I bet there is some room for improvement there.
It would be fun to be a fly on the wall when shit hits the fan in general. From Nuclear meltdowns to 9/11 ATC recordings, it is fascinating to see how emergencies play out and what kind of things go on with boots-on-ground, all-hands-on-deck situations.
Like, does Cloudflare have an emergency procedure for escalation? What does that look like? How does the CTO get woken up in the middle of the night? How to get in touch with critical and most important engineers? Who noticed Cloudflare down first? How do quick decisions get made and decided? Do people get on a giant zoom call? Or emails going around? What if they can't get hold of the most important people that can flip switches? Do they have a control room like the movies? CTO looking over the shoulder calling "Affirmative, apply the fix." followed by a progress bar painfully moving towards completion.
They said they ran a dry-run. What did that do, just generate these diffs? I would have expected them to have some way of simulating the network for BGP changes in order to verify that they didn't just fuck up their traffic.
How did no one at cloudflare think that this MCP thing should be part of the staging rollout? I imagine that was part of a // TODO.
It sounds like it's a key architectural part of the system that "[...] convert all of our busiest locations to a more flexible and resilient architecture."
25 year experience and it's always the things that are supposed to make us "more flexible" and "more resilient" or robust/stable/safer <keyword> that ends up royally f'ing us where the light don't shine.
"In this time, we’ve converted 19 of our data centers to this architecture, internally called Multi-Colo PoP (MCP): Amsterdam, Atlanta, Ashburn, Chicago, Frankfurt, London, Los Angeles, Madrid, Manchester, Miami, Milan, Mumbai, Newark, Osaka, São Paulo, San Jose, Singapore, Sydney, Tokyo."
Is the term MCP synonymous with "tier 1 PoPs" (mentioned elsewhere in other cloudflare blogs from time to time) or are the two terms referring to different things?
Something else that I think would be smart to implement is a reorder detection. Have the change approval specificy point out stuff that gets reordered, and require manual approval for each section that gets moved around.
I also think that having a script that walks through the file and points out any ovibious mistakes would be good to have as well.
Yeah, there's got to be some sweet spot between "formally verify all the things" and "i guess this diff looks okay, yolo!".
I'd say that if you're designing a system which has the potential to disconnect half your customers based on a misconfiguration, then you should spend at least an hour thinking about what sorts of misconfigurations are possible, and how you could prevent or mitigate them.
The cost-benefit analysis of "how likely is it such a mistake would get to production (and what would that cost us)?" vs "how much effort would it take to write and maintain a verifier that prevents this mistake?" should then be fairly easy to estimate with sufficient accuracy.
Would be nice to have some automation that one could use for keeping track of health status of cloud services. Status API, webhook solution, something. Maybe even a standard for it. Or a service that monitors all major cloud services.
We did get alarms. Our things partially worked though so CF was not the first thing to check.
Seems that after this outage a lot of website that are behind Cloudflare NSs now gained top positions on Google SERP with strange links like http://domain/XX/yyyyyyy
Yes, that was one of the issues they mentioned in the post. Not that they didn’t have a staging/testing environment but that it didn’t include the specific type of new architecture configuration, “MCP”, that ultimately failed.
One of their future changes is to include MCPs in their testing environments.
Actually, I think the flip side is even more interesting. If you want to give good, low latency service to 50% of the world you need a lot of data centers.
Well half of those cities were in Asia during business hours, so given that the majority of humans live in Asia it makes sense. CF data centers in Asia also seem to be less distributed than in the West (e.g. Vietnam traffic seems to go to Singapore) meanwhile CF has multiple centers distributed throughout the US.
correct, i saw that too. the outage returned 500/nginx. no version number either on footer. @jgrahamc thought that was strange too as few commenters last night were caught off guard trying to determine if it was their systems or cloudflare. supposedly its been forwarded along.
yes, there is definitely an nginx service in the path. We don't have any nginx in our infrastructure, but this was the response we had for our urls during the outage.
<html> <head><title>500 Internal Server Error</title></head> <body bgcolor="white"> <center><h1>500 Internal Server Error</h1></center> <hr><center>nginx</center> </body> </html>
It's very difficult if not impossible to create a staging environment that would well enough replicate production at this scale. What bog posts suggest as a remediation in the process:
"There are several opportunities in our automation suite that would mitigate some or all of the impact seen from this event. Primarily, we will be concentrating on automation improvements that enforce an improved stagger policy for rollouts of network configuration and provide an automated “commit-confirm” rollback. The former enhancement would have significantly lessened the overall impact, and the latter would have greatly reduced the Time-to-Resolve during the incident."
On the enterprise plans, you are able to set up your own DNS server that can route users away from Cloudflare, either to your origin or to another CDN/proxy.
Your DNS host needs to support being able to assign a CNAME record on your root domain to a domain provided by Cloudflare. AWS Route 53 does not let you do this which I imagine is a decent chunk of enterprise clients. AWS only lets you alias records to AWS resources not external domains.
With that said, even with enterprise in this case you would need to go all-in with Cloudlfare's nameservers or run the risk of not having DDoS protection on your root domain (ie. example.com wouldn't be protected but you could protect www.example.com since a CNAME with subdomains is a standard thing).
However it's kind of interesting because an attacker could get the real IP of your root domain's AWS load balancer which is probably the same load balancer used for the `www` version of your site too, but now that they know your load balancer's IP they can completely bypass Cloudflare and go straight to your infrastructure.
I'm pretty sure AWS doesn't let you assign an external domain with their aliases because they want you to pay them for AWS Shield Advanced instead of using Cloudflare because AWS knows getting an enterprise client to change their nameservers and all of their DNS records (potentially dozens of domains and multiple hundreds of records) is kind of a pain. It can be done but it's a friction point.
CloudFlare are a hosting provider and CDN, they aren't "push[ing] ... hard for recentralization of the web".
If it was AWS, Akamai, Google Cloud, or any of the other massive providers this comment wouldn't be made. I don't really understand the association between centralisation and CloudFlare, other than it being a Meme.
What do you have when you have all DNS going through them, via DoH, and all web requests going through them, if not recentralization?
Sure, they want us to think they give us the freedom to host our web sites anywhere because they're "protected" by them, but that "protection" means we've agreed to recentralize.
It's pretty dismissive to describe something as a meme just because you don't understand it, and either you're pretending to not understand it, or you truly don't.
Look at it this way: If a single company goes down for an hour, and that company going down for an hour causes half the web traffic on the Internet to fail for that hour, what is that if not recentralization?
I understand that for their WAF, DDOS and threat detection products they need to have a very large amount of traffic going through them. They have been very aggressive with their free service to achieve that, to the benefit of all their customers (including the free ones). Some could see that as a push to at centralisation, I don't.
What I don't understand, or believe, is that they want to be the sole (as in centralised) network for the internet. I don't believe they as a company, or the people running it, want that. They obviously have ambition to be one of the largest networking/cloud providers, and are achieving that.
I don't intend either to dismiss your concerns (which are a legitimate thing to have, centralisation would be very bad), my suggestion with the meme comment is that there is at times a trend to "brigade" on large successful companies in a meme-like way. That isn't to suggest you were.
They want to be a monopoly. They want everyone to depend on them. They may not want recentralization in general, but they definitely want as much of the Internet to depend on them as possible.
It's often mentioned about AWS, especially when us-east-1 fails. The others are not big enough to affect basically "the internet" when they go down, so don't get pointed out as centralisation issues as much.
And yeah, cf is trying to get as much traffic to go through them as possible and add edge services for more opportunities - that's literally their business. Also now r2 with object storage. They're already too big, harmful (as in actually putting people in danger) and untouchable in some ways.
i'm gonna go with the less popular view here that overly detailed post mortems do little in the grand scheme of things other than satisfy tech p0rn for a tiny, highly technical audience. does wonders for hiring indeed.
sure, transparency is better than "something went wrong, we take this very seriously, sorry." (although the non technical crowd couldn't care less)
only people who dont do anything make no mistakes, but doing such highly impactful changes so quickly (inside one day!) for where 50% of traffic happens seems a huge red flag to me, no matter the procedure and safety valves.
I’m surprised they did not conclude roll outs should be executed over longer period with smaller batches. When a system is complicated as theirs with so much impact, the only sane strategy is slow rolling updates so that you can hit the brake when needed.
Am I the only who really doesn't think this is a big deal? They had an outage, they fixed it very quickly. Life goes on. Talking about the outage as if it's reason for us to all ditch CF, then buy/ run our own hardware (which will be totally better), so hyperbolic.
It was a bit of a thing as people in Europe started their office work, and found out a lot of their internet services were down, and they were unable to access the things they needed. It's rather dangerous that we all depend on this one service being online.
It was very pleasant to find the cloudflare status page pointing me to the issue right away (minutes after our alerts triggered), even though I couldn't replicate the issue myself yet.
I wish more companies would take note of the transparency and sense of urgency on updating their status page. (Looking at you Azure)