Yeah, one of our goals was to not add a new/weaker point of failure.
We gracefully fall back to S3 directly if our cache server is down without a hiccup. So there is no operational overhead of this additional cog. If the server has a failure, we'd go back to slightly degraded performance by talking across the country until we brought it back online.
The idea that it being a local proxy meaning there is no operational overhead is a dangerous fallacy.
If that's an earnestly literal statement from you, then it means you simply haven't encountered the failure modes that these kinds of set up are inclined towards.
I've worked at several BigCo's, seen them all implement this pattern, and seen every single one of them have fleet-wide outages due to these innocent "local proxies".
Sure, there's definitely risk. I'm not asserting that it's literally 0 chance. But this risk of this is also tied up with other things that leverage this proxy. So it's not adding a new dependency or a new point of failure. If this has a problem, we'll also have problems talking to other services in our network.
And for what it's worth, I've definitely fucked this up in the past and caused downtime as a result of a setup like this. The pros still outweigh the cons in practice.
> You can't add a new thing without adding a new point of failure. Every point is a point of failure.
Correct, but it's an existing process. So you're right, we could ship a blatantly bad config.
> Who deploys the thing?
We do, humans, yes. We can definitely screw up a config.
> Automated deployment?
We tend to do blue/green deploys on critical pieces of infrastructure just to sanity check it. We might even pull a node out of production, test on a staging server, etc.
> Wait, was the local proxy load tested?
Yes. The load we need for this case is not even close to significant.
> Was it load tested when one of your data centers is down and everything is doing 30% more work?
Yes, it's literally just a proxy to S3 doing no additional work. For our traffic, the load is not a concern. Especially since it's running on every machine, it's distributed pretty well. A single box cannot overload our haproxy process compared to the CPU needed to run the Python application itself.
> Can you tell I used to work in monitoring? Maybe I just have PTSD now. :P
DataDog, it's pretty dope. It gives us lots of super good insight into all of these things and is what alerted us because haproxy reported S3 down in the first place. It'd also tell is the moment a process like this crashes, etc.
It's running on localhost to each server. So the failure event here is that somehow haproxy process would explode with the rest of the server being fine. It's much much more likely that a whole machine will die instead or a network issue between machines, etc.
Sure, that's well understood. Being a low-risk point of failure, isn't it still a new one? It does come with setup and maintenance costs, test scenarios etc., so it's only fair to recognise this as a risk.
Technically yes. But we're pretty accustomed to this level of risk. For something like this, the pros far outweigh the cons involved. Yeah, it could fail. The maintenance overhead of this is absolutely minimal and took a handful of hours to have tested and in production.
Also worth noting, that this isn't really a single point of failure as a system wide thing. It'd only be a single point of failure on that single node. So if haproxy decided to explode, only that one machine would have a problem momentarily, while the process got started back up with our process manager.
The worst case scenario is a human error where we ship a bad config and break everything.
This is equivalent to shipping bad application code that takes everything down. Except the config is only a handful of lines of code and will very likely never change again. Also, we don't blindly roll out changes cluster wide for things like this without testing explicitly on staging or test nodes.
Shipping it with the app, you lose the cluster wide cached objects. A SPOF is the resolver. It's google but it's a SPOF. Is the failover to s3 automatic ? Or do you make a code change ? What kind of latency does that add ?
Thx. Got it. I don't know enough about the app, but I would have it serve directly from CF to users. Instead of hitting this environment for static assets. Good job and good conversation.
We gracefully fall back to S3 directly if our cache server is down without a hiccup. So there is no operational overhead of this additional cog. If the server has a failure, we'd go back to slightly degraded performance by talking across the country until we brought it back online.