> But seems something is systematically going wrong at GH

Yes, we call it: Microslop.

It's honestly insane how terrible their reliability is. Over the past 4 weeks, 8 days with GitHub Actions outages, many of them multi-hour outages: >3 hours on each of July 9th, July 20th and today, and 1.5 hrs on July 23rd.

Outages happen, but this many outages so close together, and so many of them so major/long lasting, something is systematically wrong for sure. It's been seriously hamstringing our ability to ship code at my company.

Mon and Dad (MS) are (generally) pretty solid with uptime.

What is happening at GH?

Rate of change trying to keep up with new challengers? Over-reliance on AI? Engineers trying to debug slop?

Yeah who knows, would be interesting to hear an inside take if any readers here are also GH devs!

They're at 93.91% uptime over the past 90 days, according to https://mrshu.github.io/github-statuses/ , and that doesn't even include today's outage yet.

A glorious one nine of reliability.

I see two nines in there…

> They're at 93.91% uptime over the past 90 days, according to https://mrshu.github.io/github-statuses/ , and that doesn't even include today's outage yet.

I don’t love GitHub, but that number is a little misleading… That’s the intersection uptime of all GitHub services, most of which I (and most users) do not care about; code spaces, copilot, packages etc…

When you take out those uptimes, it becomes a lot higher. I’ll admit there seems to be a lot more incidents than usual though…

Yea just try requesting an SLA credit from them. According to THEIR numbers it’s 99.995%. It’s intolerable and I am going to put my full weight behind stripping as much work as we can from GHA as possible, even if leaving GitHub itself is effectively logistically and contractually impossible.

Don't throw shade, there are two nines in that percentage for now. Miles a part though.

911 has one! :)

> What is happening at GH?

In large part, the move from AWS to Azure. Azure's just bad.

Based on previous posts I've seen about this, IIRC the timing seems to imply that it has more to do with them being hammered with AI slop than it does with the Azure transition. Who really knows though

> ...IRC the timing seems to imply that it has more to do with them being hammered with AI slop...

It can be both things. I've worked professionally with AWS, Azure, and GCP, and Azure is just really flaky and unreliable. It's easily the worst of the three.

wasn't that years ago? or was that Skype? hahaha

AFAIK, the move to Azure is still ongoing. Rumor has it that it was delayed for a very long time after the Github acquisition because Azure was too bad to move Github to.

Azure remains too bad to move Github to, but the relevant executives have pretty clearly decided that that's no longer a good reason to delay the move.

"most of us have been there done that" - so true. That feeling in your gut when you realize something you just did caused an outage is pretty unique.

They're one rewrite in Rust away from fixing everything. (jk)

Time to touch some grass. Better use of my time and energy than twisting the remaining things on my todo list today to make more progress on them than I have managed. I should have taken a long lunch but I rebased the hell out of a PR instead.