Offshore tax havens don't turn over their records to the IRS

US based coin exchanges like Coinbase do

What prevents Coinbase to go offshore then ?

The fact that they have a ton of VC they're already beholden to that is here in the US, and who aren't interested in IRS scrutiny themselves.

Also, the fact that Coinbase is in the US legal jurisdiction gives its clients confidence they would be made whole if anything untoward were to happen.

If the untoward event is solely in USD, perhaps, but not if it's in crypto. Exit scams, hacks, rogue employees, etc, could still hit a US exchanges' crypto holdings and the government probably cannot or will not compell a roll back, supply expansion, etc, on a major blockchain.

If anything is actually protecting clients against crypto theft, it's the exchange's privately-purchased insurance rather than the government. Although I wouldn't rely on that for much either as insurers are rarely eager to pay out.

A desire to interface with the US banking system.

Plenty of offshore exchanges. People use coinbase because it's working within the US system.

Exactly this. From a consumer perspective, being subject to the US legal system is a big advantage.

They want to do business in the US and work with US banks.

That's not the kind of business they want to be. Coinbase want to be seen as legit, not an offshore tax haven.

Offshore tax havens are for the wealthy, who are largely immune to these things.

The 'wealthy' and corporations have the finances to pay for accountants and attorneys to set these things up. Most of them are taking advantage of what is legally already there, but no where near common knowledge. Wading through tax code is a struggle at best.