> I don't want to put my life in the hands of a fragile system that I, and no one alive, can understand. [...] just hope nothing out of the ordinary will ever happen while we're in the car.
This honestly sounds pretty similar to the current state of affairs.
Currently, there are 0 people that fully understand the human brain. If you're driving, and someone turns in front of you suddenly, you can ask "why did they do that", but neither you nor anyone else can truly understand the set of inputs and thoughts that went into that decision at the time. Even if the other driver says "I guess I didn't see you", they also don't understand their own brain. It's probably a post-hoc rationalization.
The argument you have applies just as well to letting humans drive as letting opaque neural networks drive... better in fact since we have a better understanding of neural nets than we do of brains.
I think this is a fundamental problem with driving cars, and it's exactly this reason that I would rather take trains and subways exclusively than drive. Even if the operator's brain has a weird malfunction on a train, we have well understood systems (tracks) that prevent them from turning into traffic.
The main difference is you have knowledge of how your own brain operates in reality, what kind of actions you might take in a given situation (and by extension the same can be said of other humans), whereas most (if not all?) people have no clue how neural nets operate, so all bets are off when it comes to predicting what the computer could/will do.
Contrast that with a human: you know the subset of possible actions a human would likely take. Even when you see a car swerving and acting like the driver is drunk, you have a mental model of the set of actions to expect from such a car (you might add more distance between you and that driver for example).
This honestly sounds pretty similar to the current state of affairs.
Currently, there are 0 people that fully understand the human brain. If you're driving, and someone turns in front of you suddenly, you can ask "why did they do that", but neither you nor anyone else can truly understand the set of inputs and thoughts that went into that decision at the time. Even if the other driver says "I guess I didn't see you", they also don't understand their own brain. It's probably a post-hoc rationalization.
The argument you have applies just as well to letting humans drive as letting opaque neural networks drive... better in fact since we have a better understanding of neural nets than we do of brains.
I think this is a fundamental problem with driving cars, and it's exactly this reason that I would rather take trains and subways exclusively than drive. Even if the operator's brain has a weird malfunction on a train, we have well understood systems (tracks) that prevent them from turning into traffic.