Contribution to the handbook
Priority · Assessed 2026-10-27
It brings together consequential barriers to citizens’ and reviewers’ ability to scrutinize systems: interconnected complexity, restricted access, and methods that may resist review. These limits bear on implementation and on whether formal review can be meaningful.
Does this challenge the goal?
In-scope gap
The question investigates practical barriers to understanding and review, which fit the existing goals of citizen service, accountability, and resilience; it does not contest those goals.
The evidence describes potential barriers, including complexity, secrecy, and limits of review, but does not measure their frequency or effects in particular systems.
Why this category?
- consequential risks and limits
- implementation dependency
- safeguards
The material is largely attributed argument and interpretation, not empirical proof that systems are unreviewable or that proposed legal or technical responses are effective.
This is an AI assessment of contribution to the goal, separate from whether the question has an answer.
- Why might people struggle to understand an algorithmic system?
- People need to examine the factors
- When systems are favoured for knowledge
- Complex systems are hard to understand
- Designers may not understand the whole system
- Understanding may be the deeper problem
- Secrecy can block inspection
- Some systems resist review
- Possible in principle, difficult in practice
- A possible public decision risk
- Checking AI and protecting viewpoints
Authority, accountability and resilience →
Making and interpreting laws →
Judging situations and resolving disputes →
Tests, stop conditions and unresolved work →
Read the exploration goalExplain the relevant barriers to understanding how an algorithmic system works, including barriers beyond mere access to it where applicable.
What we can say so far
People may struggle because algorithmic systems can be technically difficult to interpret, especially when data-mining methods are not readily understandable or when decisions emerge from interactions among many connected systems. Secrecy can also prevent inspection: companies may invoke trade-secret protections, while agencies may withhold information to prevent gaming or hacking. The difficulty may extend even to designers and managers who do not understand the whole system. Separately, people may be steered by systems without being told how they work. These are proposed mechanisms and risks in the supplied accounts, not measured estimates of how often they occur. The possibility of designing systems to support understanding qualifies the claim: the material says this may be possible, though difficult, and does not establish that the barriers are unavoidable.
What remains open
- The material does not quantify how often these barriers arise or their effects on people.
- It does not establish which disclosure or design measures would be sufficient, or whether requiring interpretable methods would preserve system benefits.
Conditions and limits
- The main accounts are interpretations of Danaher’s author manuscript, rather than independent new evidence or the publisher typeset version.
- The barriers are presented as possible or argued-for mechanisms; their prevalence and effects are not established in the supplied passages.
- The author allows that systems might be designed to support understanding, while describing practical obstacles to doing so.
Objections to consider
- The possibility of designing systems to support understanding qualifies any claim that these barriers are unavoidable. The supplied material says such design is possible in principle but difficult; it does not establish which barriers can be overcome or how effectively.
Evidence and related ideas
Supporting evidence, qualifications and context play different roles in this answer.
supports
Complex systems are hard to understand When algorithms are combined into large, connected systems, understanding how the whole system works can become extremely difficult and time-consuming.
Explains how interactions among many connected algorithms can make understanding the whole system difficult and time-consuming.
The Threat of Algocracy: Reality, Resistance and Accommodation