Tools define a relationship with the entity being tooled, so you can learn a lot about the goals and values of the transactions by looking at the kinds of relationships that are/aren't implied.
You can't feed people with an AR-15.
The metaphors that define computing are all about abstraction, automation without human input, predictability, and control.
Abstraction tends to lossy and context-insensitive, which is why an algorithm will say "no" to a loan request based on a credit score that ignores personal context, where a human loan manager who knows the people involved might say yes - for good and valid reasons.
Computing seems to be one of the most successful products of modernism, which values speedy predictable repetition, lack of decoration, and anonymised mechanisation.
It's possible to imagine computing in other value systems. But it's revealingly hard to do without losing the essence.
All of which is a complicated way of agreeing - computing is not neutral, and very probably can't be.
It's the first automated value evangelising system in history. And that's not a comfortable thought.
> Abstraction tends to lossy and context-insensitive, which is why an algorithm will say "no" to a loan request based on a credit score that ignores personal context, where a human loan manager who knows the people involved might say yes - for good and valid reasons.
A human loan manager, given only a credit score, will approve the loan based on only the credit score. The problem isn't human vs machine, it's context-awareness and context-sensitivity vs a lack of one or both.
All large systems, human or machine, tend to be implemented using abstractions. Layers of context-stripping abstractions were a part of buerocracy far before machines took it over. If a loan request form doesn't have an "if I don't get this loan my family will literally starve to death" box that you can check, the evaluator will not know about that context couldn't possibly act based on it. The form can be digital or paper and the evaluator can be a program or a human - it doesn't change a thing.
Similarly, context-sensitivity (the willingness to let the context sway your decision) was strictly limited even back in the day when decision makers were close enough to the source to be aware of the context. A bank clerk had to stick to the rules and couldn't just decide to give a loan to someone that didn't qualify under them. If they did, they'd be risking losing their job or worse, so they didn't. This discretion was only given to the people in charge - and that hasn't changed with computers. A person in a high enough position can override what the computer says, just as easily as they can override a lower clerk's decision.
Yes, making machines context-aware takes some effort, and making them context-sensitive takes even more effort. But even if machines magically came with both, their operators would do the best to get rod of them, just like they did with their employees. From their point of view, empathy is inefficiency. And that's the real problem.
Okay, but consider that I might be better off with a machine that approves or disapproves based on credit history and credit score than a banker who won't lend to me unless I get my husband's signature because he doesn't think women should be making choices about money without their husband's approval. (But who wouldn't even blink at loaning money to a man without his wife's signature.)
You can't feed people with an AR-15.
The metaphors that define computing are all about abstraction, automation without human input, predictability, and control.
Abstraction tends to lossy and context-insensitive, which is why an algorithm will say "no" to a loan request based on a credit score that ignores personal context, where a human loan manager who knows the people involved might say yes - for good and valid reasons.
Computing seems to be one of the most successful products of modernism, which values speedy predictable repetition, lack of decoration, and anonymised mechanisation.
It's possible to imagine computing in other value systems. But it's revealingly hard to do without losing the essence.
All of which is a complicated way of agreeing - computing is not neutral, and very probably can't be.
It's the first automated value evangelising system in history. And that's not a comfortable thought.