I’m sure some retries are helpful there too (TCP is doing them, at the very least) but yeah, different approaches for different situations. Maybe you retry but you don’t spend many seconds hoping for it to work.
I'm ok with TCP retries generally. But the assumption at L4 is that L3 can and will drop/reorder packets at random, and retrying is cheap. Also a lottt of tuning has gone into TCP already.