> It's not actually in their interest to look too closely. They know they are selling snake oil, and they do not care if it works.
I can tell you from direct experience on the inside at AWS that they care tremendously about virtualization security.
It's why they never let two tenants share a physical core simultaneously; they were prepared against side-channel attacks before Spectre and Meltdown were discovered.
It's why they never supported Intel SGX, preferring instead to sequester processors for confidential workloads (Nitro Enclaves), and why modern EC2 instances support memory encryption (Graviton + AMD SEV-SNP).
It's why, when you improperly remove a chassis from its cabinet, a mechanism destroys a key component containing the machine's attestation and encryption keys, rendering it useless.
It's why they delegated a large amount of security surface, including practically all I/O functions, to their off-CPU Nitro card.
> For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)
Why do you say it's not supposed to happen? It's the predictable outcome of not having unlimited scale. The hyperscalers are big, but to be able to accommodate a tectonic shift in demand in a very compressed amount of time is very costly--costs that will be passed to their customers--if it can be done at all given current constraints. Today's hardware crunch, even for hyperscalers, is very real; customers are hoovering up spare capacity like never before.
As a former guidance counselor (specialist SA) for AWS, I regularly told customers that they shouldn't expect to be able to pack up and relocate into a new AZ once an incident commenced. They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.
> They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.
Do you blame them? You not only have to pay double for the infra but also be charged for inter-region data transfers. And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
1. AWS has no cross-regional or cross-zonal data plane[1] dependencies. Customers might couple them, in violation of best practices, but AWS itself does not. It's strictly forbidden.
2. That article you linked to does not support your claim. That is a report of a single-zone incident impacting us-east-1 only. Five other AZs in the same region were still fully operational.
[1] These are not the same as control plane dependencies: you might not be able to make certain mutations, but everything should still work in the steady state.
While what you say is generally true for 1, I wholly disagree that it is always customers that do the coupling.
One more than one occasion there are AWS URLs, including redirects inside the console, that include US East 1 in the URL that were only spotted during an outage. Manually changing it to an different region allowed us to get around it but to wholely blame customers when you don't know the context is bad form.
But there are also always exceptions. I've seen US East 1 outages cause autoscaling issues in other regions including failing to start new EC2s. A real problem if things have scaled down overnight.
> One more than one occasion there are AWS URLs, including redirects inside the console, that include US East 1 in the URL that were only spotted during an outage. Manually changing it to an different region allowed us to get around it
Yeah, that tracks. The Console has often been wonky like that.
> I've seen US East 1 outages cause autoscaling issues in other regions including failing to start new EC2s
Do you mean scaling failed due to insufficient capacity, or something else? ICE errors I get (thundering herd of attempted capacity shifting); but otherwise, that sounds implausible, and I respectfully believe you are mistaken about cause and effect. Happy to evaluate a causal chain of evidence that proves you correct, though.
My documentation for events from 2018 at this point are limited, and trying to go back through the lists of events demonstrates the limited reporting for some things deemed "non-critical" such as the early 2019 connectivity issues with EU West 2 when the London fibres were cut and services such as Workspaces experienced an outage.
I will do better at saving exact details for internet strangers in future.
reply