It's more like you'd quit a company you like if your manager is making your day-to-day life miserable (ish).

A good mansger miiight be able to retain you longer under bad external conditions because they shield you from them.