This can either be a relatively short time or an eternity, depending on the upstream consequences. If that means holding a connection open for 60000 milliseconds while waiting for some downstream rpc to go through its backoff ritual, that's an eternity, and under load that connection pool will get exhausted quickly. So now a problem which should only affect maybe 1% of users has completely hosed everyone.
I've seen this happen multiple times. Someone designs some clever backoff strategy without considering how it fits in the context of the rest of the system. Hilarity ensues.
If you have something taking an entire minute on a computer, please ensure you implement it in such a way that no connections are actually held open for that entire minute.
With http you can do it elegantly with a 429 and retry-after. 100% doing it at the edge is the way to go. The machinery which determines how long to delay a client's retry can benefit from knowledge of internal services' state, but I agree the ultimate decision must lie with the serving layer. That's the only way to efficiently deal with misbehaving clients.
On that note, one of the more memorable incidents of my career was when a 10M+ node client decided to retry as hard as possible on 4xx. That was fun x_x.
[edit] that is to say, for this mechanism to be robust your retry-after enforcement mechanism needs to be capable of withstanding almost every single one of your users attempting to illegally retry as fast as they physically can without negatively impacting that one user requesting legitimate traffic. https://media.tenor.com/p3mss3YI6TcAAAAM/wat.gif
I've seen this happen multiple times. Someone designs some clever backoff strategy without considering how it fits in the context of the rest of the system. Hilarity ensues.
If you have something taking an entire minute on a computer, please ensure you implement it in such a way that no connections are actually held open for that entire minute.