Contribution to the handbook
Priority · Assessed 2026-10-28
The evidence reports a concrete reliability risk: predictions could change when option order was reversed, with consistency below 70% in some cases. That makes order reversal a consequential test condition for a system using models to represent preferences, without establishing that the finding generalizes.
Does this challenge the goal?
In-scope gap
Testing whether predictions change when options are presented differently is a reliability question within the existing goal of designing and testing AI governance; it does not challenge that goal.
The evidence concerns preference predictions in one study, not models used in governance functions.
Why this category?
- Identifies an observed answer-instability risk with a direct implication for reliability testing.
- The reported variation across models and cases is material; the evidence does not establish its prevalence in other settings.
The supplied records give no exact rates for every model or case and do not establish how often the inconsistency occurs beyond the study.
This is an AI assessment of contribution to the goal, separate from whether the question has an answer.
Authority, accountability and resilience →
Tests, stop conditions and unresolved work →
Read the exploration goalAssess evidence of answer consistency under changes such as reversing option order, distinguishing observed inconsistencies from a claim about all models or choices.
What we can say so far
The evidence indicates that answers are not uniformly consistent when option order changes. The study defines consistency as the share of pairwise choices that remain the same after reversing the two options. GPT-3.5 Turbo and Falcon 7B were generally consistent in more than 70% of cases, while the other models did not reach that level despite similar accuracy. The reported results are model- and case-specific; they do not establish that every model or choice is inconsistent, or that any model is always consistent.
What remains open
- Exact rates for each model and sample size are not provided.
- The excerpts do not establish how frequently inconsistency occurred across all tested cases.
Conditions and limits
- The reported greater-than-70% consistency applies generally to GPT-3.5 Turbo and Falcon 7B, not necessarily to every case.
- The excerpt does not provide exact model-by-model rates or show how rates varied by sample size.
- The source summaries derive from the same preprint and are not independent verification.
Objections to consider
- Including both option orders was intended to address order sensitivity, but the passage does not establish that this method eliminated or reduced inconsistency.
Evidence and related ideas
Supporting evidence, qualifications and context play different roles in this answer.
supports
Changing option order could change answers The authors report that model predictions could be unclear and could change when the order of two options was reversed. In some cases, consistency was below 70%.
Reports that predictions could change when option order was reversed and that consistency was below 70% in some cases, while limiting this finding to some cases.
Large Language Models (LLMs) as Agents for Augmented Democracy
Read full source: Large Language Models (LLMs) as Agents for Augmented Democracy (opens in a new tab) ↗Read source discussion: Large Language Models (LLMs) as Agents for Augmented Democracy →