Strong agree. We were using Fargate nodes in our us-east-1 EKS cluster and not all of our nodes dropped, but every coredns pod did. When they came back up their age was hours older than expected, so maybe a problem between Fargate and the scheduler rendered them “up” but unable to be reached?
Either way, was surprising to us that already provisioned compute was impacted.
Saw the same. The only cluster services I was running in Fargate were CoreDNS and cluster-autoscaler; thought it would help the clusters recover from anything happening to the node group where other core services run. Whoops.
Couldn't just delete the Fargate profile without a working EKS control plane. I lucked out in that the label selector the kube-dns Service used was disjoint from the one I'd set in the Fargate profile, so I just made a new "coredns-emergency" deployment and cluster networking came back. (cluster-autoscaler was moot since we couldn't launch instances anyway.)
I was hoping to see something about that in this announcement, since the loss of live pods is nasty. Not inclined to rely on Fargate going forward. It is curious that you saw those pod ages; maybe Fargate kubelets communicate with EKS over the AWS internal network?
I'm a software engineer specializing in building stable, scalable consumer-facing websites and refactoring large codebases. Looking for consulting or contract work, working solo or with a team. Some previous projects:
- Built a comprehensive CI ecosystem for a 30+ app polygot environment using Jenkins, Docker and Github. Reduced test suite run time in their monolith from 25 minutes to 10.
- Refactored the real time auction draft system for a fantasy sports company to increase stability and support four times as many users without additional hardware.
Indeed - I've hiked almost 3k miles carrying an umbrella. Worn rain gear is a compromise between breathability and waterproofness - even if you keep the rain out, you'll sweat and it's trapped inside. Umbrellas keep rain off you, are very breathable, and double as sun protection when it's really hot. Only downside is when it's windy and rainy, so it's nice to have a poncho too.
I'm a software engineer specializing in designing and building stable, scalable consumer-facing websites and refactoring large codebases. Currently, I'm helping clients with dev ops problems like setting up web apps to scale for 10x more traffic, profiling/optimizing slow code paths, and building analytics pipelines with AWS.
Before that, I was a principal engineer at Vox Media, and worked on some notable projects:
I'm a software engineer specializing in designing and building stable, scalable consumer-facing websites and refactoring large codebases. Currently, I'm helping clients with dev ops problems like setting up web apps to scale for 10x more traffic, or building analytics pipelines with AWS.
I'm a software engineer specializing in designing and building stable, scalable consumer-facing websites and refactoring large codebases. Currently, I'm helping clients with dev ops problems like setting up web apps to scale for 10x more traffic, or building analytics pipelines with AWS.
Before that, I was a principal engineer at Vox Media, and worked on some notable projects:
Either way, was surprising to us that already provisioned compute was impacted.