I also can't pretend to be an expert on this. Conceptually, everything you're saying makes sense.
It still just sounds like a really vanilla "horizontally scaling VMs" problem, especially for yesterday's incident that was focused on pipelines.
If this is a "GitHub is giving away more capacity than it has" problem, that's easily solved by rate limiting and queuing.
This is GitHub's explanation:
> During the incident, some Actions Runner Controller (ARC) runner pods became stuck in an idle state. Affected users can delete those pods using kubectl or redeploy their Actions Runner Controller application. ARC will automatically create replacement runners.
> The next releases of Actions Runner and Actions Runner Controller will include an automatic recovery mechanism, preventing the need for these manual steps in the future.
That sounds a lot more like a major architectural flaw and a really obvious oversight.
It still just sounds like a really vanilla "horizontally scaling VMs" problem, especially for yesterday's incident that was focused on pipelines.
If this is a "GitHub is giving away more capacity than it has" problem, that's easily solved by rate limiting and queuing.
This is GitHub's explanation:
> During the incident, some Actions Runner Controller (ARC) runner pods became stuck in an idle state. Affected users can delete those pods using kubectl or redeploy their Actions Runner Controller application. ARC will automatically create replacement runners.
> The next releases of Actions Runner and Actions Runner Controller will include an automatic recovery mechanism, preventing the need for these manual steps in the future.
That sounds a lot more like a major architectural flaw and a really obvious oversight.