A starting point for discussion
The idea
The paper argues that decision systems based on algorithms—a set of rules a computer follows—may be especially likely to become hard to understand. People might first favour them for apparent gains in speed, accuracy, and ability to see patterns or draw conclusions, then come to rely on them without being able to understand how they work.
Place in the reasoning
AI assessment · Connected to reasoning
This describes a possible pathway from apparent performance benefits to reliance without understanding, not a claim that every system follows it.
Computer systems can shape what people do without taking over
It gives a possible way systems valued for speed, accuracy, or insight may become relied upon despite limited understanding.
Adapting to computer systems or resisting them can both carry risks
The appeal of apparent benefits helps explain why resistance and accommodation involve trade-offs.
What this means and why it matters
What this means
A system may gain support because it appears to perform well compared with a human alternative. As people rely on it, its workings may become harder for them to follow.
Why it matters
A system’s apparent usefulness does not by itself show that people can understand or assess its decisions.
Conditions & assumptions
Source coverage: Full readable text of the author’s manuscript from an accessible mirror; this is not the final version formatted by the publisher. The author presents a possible gradual path from a system’s apparent usefulness to growing difficulty understanding how it works, not a finding that every such system follows it. Addresses: The paper’s account of the risk of rule through algorithms
Questions this connects to
AI-suggested relationships. Authors have not necessarily responded to one another.