As of 2020, only 1% of physicians opted out of Medicare[0]. That is an increase of only 0.3% since 2013. Furthermore, in my experience, physicians that opt out of Medicare refuse to accept any insurance, so the number of Medicare opt-outs is probably more of a proxy for opting out of insurance entirely. It is really not “a lot”.

[0] https://www.kff.org/medicare/issue-brief/how-many-physicians...