It's a pretty serious psychological barrier. When you already own a car, the maintenance and finance costs are already sunk costs, so the marginal cost of doing a weekend trip is quite low -- pretty much just the cost of gas. When you don't own a car, now you have to throw in the cost of the rental too, which can easily make a $20 trip turn into a $120 trip or more (it costs $100 per day to rent a car near me, so an weekend camping trip jumps up to $220). Even if the total cost with the occasional rental is less, the marginal costs are structured in a completely different way, which ends up negatively incentivizing it.

Hell, I live in Manhattan and I now don't do a lot of things I used to do when I lived out in the suburbs and had a car, for precisely this reason. Instead I do a lot more things on my bicycle, including long weekend trips -- it's basically become to me what my car used to be to me when I lived in the suburbs. Except using it actually improves my health.