A starting point for discussion
The idea
Human review may be impossible when a system uses methods that search large amounts of data for patterns that people cannot interpret, or when a decision only makes sense as part of a wider group of connected systems. The author notes that a law could require methods people can interpret, but the passage leaves the consequences of that choice open.
Place in the reasoning
AI assessment · Connected to reasoning
This identifies technical conditions that may make human review impossible, including uninterpretable methods and decisions dependent on connected systems.
People need to understand decisions before they can check them
These possible barriers support the concern that formal monitoring or override does not guarantee capable human review.
Proposals to improve people’s abilities and require human review have limits
A legal requirement for interpretable methods is raised as a possibility, but its consequences are left open; this qualifies rather than resolves the proposal.
What this means and why it matters
What this means
A person may not be able to understand how a system reached a decision, either because its method is hard to interpret or because the decision depends on many linked systems.
Why it matters
A right to review may offer little practical help if people cannot access or make sense of the reasons behind a decision.
Conditions & assumptions
Source coverage: Full readable text of the author manuscript from an accessible mirror; not the publisher typeset version. The author gives two reasons that technology may block review and raises, without resolving, the option of requiring methods people can interpret.
Questions this connects to
AI-suggested relationships. Authors have not necessarily responded to one another.