A surprising number of humans have no problem with doing things to faceless computers and public or corporate infrastructure that they would never do to another human being with whom they were making eye contact.
Few are willing to don the reflective vest and ruin someone's day by lying to their face and directing them down a bad detour, but I have no problem believing that a lot of people would laugh at a stupid self-driving car wasting its time and energy following their hand signals.
But a self driving car SHOULD follow anyone directing traffic.
There are valid reasons for <random citizen> to be directing traffic, like losing their load on the highway or helping traffic flow around a stopped vehicle.
Would you step in front of a self-driving car to make it emergency brake? I think a lot of people would if they could. It's another instance of what you mentioned - teenage gangs bullying and playing with lonely empty self-driving cars.
Few are willing to don the reflective vest and ruin someone's day by lying to their face and directing them down a bad detour, but I have no problem believing that a lot of people would laugh at a stupid self-driving car wasting its time and energy following their hand signals.