You will not know how safe they are until accidents happen. I would love to see someone with money run an experiment with real drivers vs google cars. Create a worst case scenario course with simulated drunk drivers, blown tires, missing stop signs, etc.. and run this with real and fake drivers. I suspect the average driver would crash 10% of the time, autonomous car would crash 1% of the time, and a professional driver would crash never.
The real safety improvements will come when all cars are autonomous and they talk to each other.
"Missing stop signs" is one place where a google car would be great, since they pre-map the area and would know a stop sign has disappeared. Knowing that would cause them to slow down and be even more careful.
Autonomous cars never get distracted, have 360 degree vision, look at all things all the time, and can react in tens of milliseconds versus hundreds of milliseconds. I'd trust that over a "professional driver."
The real safety improvements will come when all cars are autonomous and they talk to each other.