I live in (ok, ok, near) Boston and I constantly fantasize about this. I don't commute by car I look forward to riding my bike 5mph in front of driverless cars, at least if they don't run me over...
I also have to imagine they will be a ripe target for vandalism, especially if operators don't aggressively remove graffiti.
And they'll also be surrounded by camera's and hard drives. I would imagine that the sensors that can see through fog could also get more detail than the average CCV camera - meaning that even with a mask, you're going to get a lot of detail on vandals or road hazards (aka: You going 5mph in front of one)
Vandalism already happens to cars. As are cyclists who go 5mph in front of a car (on a road where it cannot overtake you) to take revenge on a driver they don't like.
None of that will change and that's not dependent on drivers.
I also have to imagine they will be a ripe target for vandalism, especially if operators don't aggressively remove graffiti.