I think physical violence is qualitatively different.
Consider: suppose a kid lays across your driveway and refuses to budge. Are you allowed to drive over the kid? After all, if everyone starts doing this, you will lose your job and can't feed your own family, right?
Well, no. You have to figure out a way. Take a deep breath and call their parents, get an Uber or whatever it takes.
The situation would be different if a random person was just asking you not to go. In that case, you could ignore them.
Well, in your example, I can't run over the kid, but I can call the cops, and have them come and physically remove him. So that's almost exactly what happened, except that the burly guy wasn't a cop (or at least wasn't in uniform).
The cops were there, though, and were (at least tacitly) authorizing the guy to try to physically remove the passenger. Is that legal and/or moral? Or does it have to be a uniformed officer before it becomes legal and moral?
It's easy for a couple of cops (or even one) to remove a kid from your driveway, even a kid who doesn't want to be removed. Removing an adult from what appeared to be a window seat is going to be harder. That is, the adult can make it harder to do without causing harm to the adult. (Also, the adult is presumably more aware of the potential consequences of their actions.)
You have a "contract of lying on the driveway" with him, you've granted him permission to lie on the driveway for the next 1.5 hours, and your contract doesn't list you changing your mind as a reason for rescinding that prior permission, nor has he breached the contract in a way that permits you to remove him.
You're welcome to point out where in the contract of carriage you're basing your opinion. I'm finding no justification.
Consider: suppose a kid lays across your driveway and refuses to budge. Are you allowed to drive over the kid? After all, if everyone starts doing this, you will lose your job and can't feed your own family, right?
Well, no. You have to figure out a way. Take a deep breath and call their parents, get an Uber or whatever it takes.
The situation would be different if a random person was just asking you not to go. In that case, you could ignore them.
Because physical violence is different.