>> but people always gloss over the fact that we really don't know much at all about how the human brain generates its results.
If you look carefully, you'll see that interpretability is an issue in domains
with either safety or liability requirements. Sometimes both are required-
like in medicine where a doctor must be able both to make decisions that do
not endanger their patient and justify those decisions if something goes
wrong.
In such domains, all technology tends to have very high reliability stnadards, in the sense that it is made so that it's possible to make predictions about its behaviour under various assumptions.
With neural nets, the black-box nature translates to a lack of
predictability, which is unacceptable.
Btw, in the same kind of domains, human introspection seldom comes into play.
Human actors are normally trained professionals who can explain their actions
with minimal recourse to introspection. For instance, a doctor will tell you
"I administered 50cc of Chloraxine because it has been shown to be beneficial
in cases where the patient suffers from third-degree burns etc etc", not "I
felt ambitious and bold and decided to give the patient some Chloraxine".
If you look carefully, you'll see that interpretability is an issue in domains with either safety or liability requirements. Sometimes both are required- like in medicine where a doctor must be able both to make decisions that do not endanger their patient and justify those decisions if something goes wrong.
In such domains, all technology tends to have very high reliability stnadards, in the sense that it is made so that it's possible to make predictions about its behaviour under various assumptions. With neural nets, the black-box nature translates to a lack of predictability, which is unacceptable.
Btw, in the same kind of domains, human introspection seldom comes into play. Human actors are normally trained professionals who can explain their actions with minimal recourse to introspection. For instance, a doctor will tell you "I administered 50cc of Chloraxine because it has been shown to be beneficial in cases where the patient suffers from third-degree burns etc etc", not "I felt ambitious and bold and decided to give the patient some Chloraxine".