Contribution to the handbook
Priority · Assessed 2026-10-28
The question surfaces option-order sensitivity as a material limit on treating model predictions as stable expressions of preferences. The evidence describes a consistency measure across models and sample sizes, reports that GPT-3.5 Turbo and Falcon 7B generally exceeded 70% consistency, and notes that other models did not reach that level despite similar accuracy. This supports reporting a consequential failure mode and testing limitation, not a claim that any model is fit for governance.
Does this challenge the goal?
In-scope gap
The question addresses a reliability limitation and test relevant to preference-prediction proposals within the existing goal.
The packet reports broad patterns but not exact rates for every model or sample size.
Why this category?
- Reveals a consequential reliability risk in a proposed preference-prediction mechanism.
- Provides a specific test measure and comparative result.
- Supports testing and qualification of accuracy claims.
The supplied results do not give exact model-by-model rates or establish how order sensitivity would affect real governance decisions.
This is an AI assessment of contribution to the goal, separate from whether the question has an answer.
Authority, accountability and resilience →
Tests, stop conditions and unresolved work →
Read the exploration goalExplain how consistency was measured across models and sample sizes, and report the stated pattern of which models did or did not generally retain choices.
What we can say so far
The authors defined consistency as the share of pairwise choices that stayed the same when the two options’ order was reversed. They report comparing this rate across models and different sizes of probabilistic samples. In their broad pattern, GPT-3.5 Turbo and Falcon 7B retained choices in more than 70% of cases; the other models generally did not reach that level, despite similar accuracy. The authors also report that consistency could be below 70% in some cases. These are reported findings from the paper, not independently interpreted figure results.
What remains open
- Exact consistency rates for each model are not supplied.
- The supplied material does not specify the sample sizes or how consistency rates varied with sample size.
- The passage reporting below-70% cases does not specify their frequency or which models and cases they involved.
Conditions and limits
- The results are reported by the authors for the described study setting.
- The supplied material does not independently interpret the figures.
Objections to consider
- The reported below-70% cases qualify the overall result rather than contradict the model-level pattern: the main finding distinguishes GPT-3.5 Turbo and Falcon 7B from the other models and does not claim that every prediction was stable.
Evidence and related ideas
Supporting evidence, qualifications and context play different roles in this answer.
context
Six fine-tuned models The authors fine-tune six existing language models on the volunteer data: Llama-2 7B, LLaMA-3 8B, ChatGPT 3.5 Turbo, Mistral 7B, Falcon 7B, and Gemma 7B.
Identifies the six fine-tuned models in the study, but does not give their option-order consistency results.
Large Language Models (LLMs) as Agents for Augmented Democracy
Read full source: Large Language Models (LLMs) as Agents for Augmented Democracy (opens in a new tab) ↗Read source discussion: Large Language Models (LLMs) as Agents for Augmented Democracy →