But there are reasonable limits on those private property rights that are commonly accepted.

For instance, I don't have the right to set up a nightclub without permission from the city, and (if I'm not being an asshole) neighbours and other nearby residents/businesses that would be affected.

These limits come into existence because there was a problem that residents of the city felt needed action on. Leaving large numbers of properties empty is the extreme opposite end of everyone running nightclubs.