> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.
This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling failures)", but rather "this complex system collapses (rather than degrade gracefully) when demand exceeds capacity".
When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.
Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections: https://sre.google/sre-book/handling-overload/ Most other large internet services have since copied them, afaik.
My last three employers refused to take advantage of Kubernetes PriorityClasses and agree to schedule work to agree (as a cluster-wide resource that affected many teams) on what our PriorityClasses should be and to migrate workloads to have priorities. And this is something relatively easy to implement - no developer work required, and practically no YAML to write.
Why not? Because sadly, fundamentally, most workplaces are not run by people who care about day-2 operations or long-term health. Product or Sales pushes customer-visible work into the pipeline, and you dare not say no. "Day-2" work is not considered to be something that moves the needle. Even now, with GitHub facing these severe outages, it's not like they're facing some massive exodus; their load seems to be getting worse over time, not better.
I'd be very surprised if there weren't any employees at GitHub who had read the SRE book. I'd expect that they're just not listened to.
What TP is talking about has nothing to do with workload preemption and is more of a variation of loadshedding (e.g overload management in Envoy). When I was part of the team that ran Google's clusters we had very few priority classes - basically just one for system and majority of serving workload ran on another priority and the rest was for batch. Pretty sure SRE book recommends just that.
Workload pre-emption is a form of load-shedding - you shed the load of lower-priority workloads (by evicting their Pods) to free up capacity to schedule more Pods of higher-priority workloads that were added by the Horizontal Pod Autoscaler.
> basically just one for system and majority of serving workload ran on another priority and the rest was for batch
RCA blames in-house load-balancing services (HAProxy) that reached capacity limits. Even if autoscaling is not working correctly because it didn't take Istio into account - why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
Yes, just absolutely crazy way of doing load shedding.
> why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?
The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario.
> Application backends?
I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only workloads continue to succeed.
First of all, web/api being down is not the worst case scenario - i didn't even notice this till late in the day. Secondly, even 1M qps at the edge is tiny amount of compute, what you have here is not a scaling problem it's a misconfiguration problem. Don't make it worse by adding more load on the system with your preemption churn and maybe taking out more important services causing more severe/longer outage - just fix it in the edge.
are these separate services? Are you going to integrate how services preempt into overall system design?
My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applying much coarser logic to what gets squeezed.
> you want to move these decisions into the application tier as much as possible
I actually agree, but this is a luxury that most large companies cannot politically prioritize (it is not Product/Sales-driven, see earlier comment). Especially when the company is large, and there are dozens if not hundreds of developer teams in a polyglot microservice environment, pushing application-level handling of these concerns is virtually impossible without executive support, and because it doesn't move the bottom line in an easily measurable way, you won't get executive support.
Companies much prefer infrastructure-based solutions to these problems, even if they're coarser, because the relatively small number of people who need to be involved makes it politically feasible. Easy example off the top of my head - mutual TLS encrypting east-west traffic has been implementable at the application layer for decades, but it was a pipe dream until service meshes made it easy to deploy (it's still a pipe dream for many orgs that refuse to schedule any infra work not Product/Sales-driven though).
This is an excellent book and its lessons saved my bacon many times!
As it happens I have a hardcopy of this book (along with "Seeking SRE" and the "SRE Workbook") that I am giving away (because of a move). If you want a hardcopy then email me your UK address I will be happy to post them to your for free.
I tried putting them on the street in a little box but surprisingly none of my neighbours grabbed any of my software books. :)
actually, I think this time it is the other way around, Azure is more the solution than the problem.
Github in the past ran on their own Hardware. That is fine, if your load is predictable and nto changing rapidly. However, the evolution of the past few months/years has shown, that the previous assumptions about growth are now outdated and scaling that capacity on your own metal is not that easy. Hardware has lead times of many weeks, especially in the current situation, datacenter capacity is even longer and more difficult, especially right now.
Choosing not to deal with scaling the hardware is a valid choice in this situation. Yes, Azure is a bunch of servers held together with glue, duct tape and a lot of hope, but I think, the github hardware is not much better at the moment.
Agreed on not hosting things yourself, but, I am not arguing against cloud hosting at all - just Azure. If they just came out and admitted it is a disaster and swallowed their pride and moved their services towards anything else at all - GCP, AWS or whatever else - I think their uptime would significantly improve. Of course, the fundamental problem here is the culture of the company itself. That's harder to fix.
It's wild to me that when I think MS I think this article, Vista, Windows Phone, Zune, Metro, ads in the start menu, the pleated pants meme and so-on, but I have coworkers that are total evangelists for MS technologies and platforms. Clearly a lot of devs get value from the ecosystem.
A lot of their platforms (not Azure) used to just work. Visual Studio and .NET were solid. The Windows NT line was pretty good for servers too (but different from Linux, and not as flexible by virtue of not being a bazaar). It was at Windows 8 era they turned it into shit, first by designing a UI for tablets (fine) and then putting that UI on desktops (wtf) and silently upgrading Windows 7 users without their consent.
I remember running IIS and Apache side-by-side many years ago, and oh boy, was IIS a pain.
Instead of a single config dir where you just deploy the configs and you are done, IIS settings were all over the registry. And don't forget component registration - a single, global registry per machine, used by IIS. Want to run staging and dev server on same machine? Tough luck. Random software installs can upgrade components and cause failures in completely unrelated failures...
The article doesn't mention any Azure outages, but a client had an Azure hosted container restart that day which caused problems due to configs being stored out of persistant storage.
If you load an issue page, you'll see 1 failed request to:
/project/product/issues/<number>
And sure, that's what you care about, but consider the working requests to:
/in-product-messaging/copilot-budget-request-banner
/in-product-messaging/code-scanning-ai-findings-preview-banner
/github-copilot/chat
/_private/browser/stats
If page has 15 requests and needs data from all of them to work correctly, then with 20% failure rate you are suddenly close to 100% non-functional page from the user perspective.
A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, rather than 20% of traffic across all users (causing essentially full failure for all users).
We typically implemented this by propagating an ID of the parent request down to the backend. I'm simplifying a lot in this description (e.g. have to deal with the parent requests landing on different backend tasks; also rotate the IDs gradually to introduce some fairness).
A nice technique to learn of this situation is to fail ("black hole") random resources in synthetic loads of the frontend. This lets you find out 'bout these single points of failure, and then you can start adding fallbacks.
It's easier to create a nice worst-case failure UX this way too. Rather than a page that sorta kinda loads but doesn't actually work at all you can unambiguously put up a "oops we're broken right now try again later" page whenever that top-level endpoint returns a non-retryable error.
But that of course begs the question--why is the frontend retrieving all these data from 16 different endpoints instead of just one? It would be less overhead (http, auth, serialization, compression) to just serve it all in one request even if the http layer needs to fan out 15 rpc calls... this is why we can't have nice things.
Shyeah, all those successful requests were really helpful for showing me a page, which then failed when I tried to do anything by clicking the buttons on the page. I was completely unable to merge things for hours, for no lack of trying. GH was down.
Thank you so much for this gem, this sent me through the rabbit hole and I'm fascinated by the level of engineering in this book. I love how big complex systems are engineered. It feels like anything is possible when you have a solid plan and you keep improving it step by step. So inspiring.
Of course there's infinite capacity. A google datacenter is infinite capacity from the perspective of say an NTP server. Infinites exist when you have enough orders of magnitude in the middle.
Also traffic isolation and degradation by tier is not "no outage", you're still in outage land, you're just being smart in how you use it and choosing what you disrupt. It doesn't fix the lack of capacity.
Google's (or any similar large company's, really) efficiency requirements would never allow for such levels of waste. It would be incredibly irresponsible to allow just any random small server to unexpectedly use the entirety (or even just a non-trivial percentage) of a Google-scale datacenter's capacity. Do you evict other jobs? Do you run your datacenters cold to prepare for this eventuality? Both are terrible options.
Suddenly you push an update to your tiny server or one of its thousands of clients and, because of some bug or misconfiguration, traffic or per-request-resource-consumption grows 1000X or more. Given the size of the looong tail of small servers at Google, this would be a daily occurrence.
I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding.
And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, but it significantly lessens the negative effects of it.
You understand it if you think of engineering infinites rather than mathematical infinites. The capacity of a full single datacenter can be treated as infinite for most customers. I explained how its defined in the original comment. Amount of places in engineering where you treat even a 3 order of magnitude difference as infinite is a lot, but the number of order of magnitudes varies depending on context.
You don't have a single Google datacenter (or any other provider / big name) for a single tenant. The bigger the DC, the more the number of tenants/customers reaches "infinity" (giving the same meaning to "infinity" you are giving to it)
Sure there is infinite capacity given an infinite amount of time to scale up. I assert you’re leaving out the time dimension. Those so-called infinities are simply not accessible in a practical way since you’ll hit a wall in actually provisioning that capacity long before the data center runs out of compute.
> this complex system collapses (rather than degrade gracefully) when demand exceeds capacity
This.
It's easy to be an "armchair quarterback," here, though.
Handling stuff like this, needs to be planned for, from the start. I suspect that a lot of the issues are because GitHub is something that started small (and probably quickly), and has accreted. Things like Facebook are in a similar boat.
This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling failures)", but rather "this complex system collapses (rather than degrade gracefully) when demand exceeds capacity".
When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.
Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections: https://sre.google/sre-book/handling-overload/ Most other large internet services have since copied them, afaik.