Sure, there's definitely risk. I'm not asserting that it's literally 0 chance. But this risk of this is also tied up with other things that leverage this proxy. So it's not adding a new dependency or a new point of failure. If this has a problem, we'll also have problems talking to other services in our network.
And for what it's worth, I've definitely fucked this up in the past and caused downtime as a result of a setup like this. The pros still outweigh the cons in practice.
> You can't add a new thing without adding a new point of failure. Every point is a point of failure.
Correct, but it's an existing process. So you're right, we could ship a blatantly bad config.
> Who deploys the thing?
We do, humans, yes. We can definitely screw up a config.
> Automated deployment?
We tend to do blue/green deploys on critical pieces of infrastructure just to sanity check it. We might even pull a node out of production, test on a staging server, etc.
> Wait, was the local proxy load tested?
Yes. The load we need for this case is not even close to significant.
> Was it load tested when one of your data centers is down and everything is doing 30% more work?
Yes, it's literally just a proxy to S3 doing no additional work. For our traffic, the load is not a concern. Especially since it's running on every machine, it's distributed pretty well. A single box cannot overload our haproxy process compared to the CPU needed to run the Python application itself.
> Can you tell I used to work in monitoring? Maybe I just have PTSD now. :P
DataDog, it's pretty dope. It gives us lots of super good insight into all of these things and is what alerted us because haproxy reported S3 down in the first place. It'd also tell is the moment a process like this crashes, etc.
And for what it's worth, I've definitely fucked this up in the past and caused downtime as a result of a setup like this. The pros still outweigh the cons in practice.