Contribution to the handbook
Priority · Assessed 2026-10-28
The held-out accuracy results and comparison with a party-orientation rule provide a concrete test result relevant to whether models can predict individual choices. They do not establish adequate representation, transfer to other populations, or performance in a real governance system.
Does this challenge the goal?
In-scope gap
Individual-choice accuracy and its benchmark are evidence relevant to testing a proposed civic AI use; the question does not challenge the goal itself.
The test is limited to the study’s participants, proposals, models, and comparison methods.
Why this category?
- testing
- consequential limits
- meaningful benchmark
The supplied results come from one study and shared source material. They do not show performance outside the study or establish that prediction is equivalent to representing a person.
This is an AI assessment of contribution to the goal, separate from whether the question has an answer.
Tests, stop conditions and unresolved work →
Authority, accountability and resilience →
Read the exploration goalReport the models’ individual-choice accuracies, including the test setup and benchmark needed to interpret them.
What we can say so far
The reported test trained on choices from 50% of participants and evaluated predictions on choices from the other 50%; the study describes participant-held-out cross-validation. The data comprised 8,719 pairwise choices from 267 participants across 67 proposals. Reported individual-choice accuracy was 76.68% for LLaMA-2 7B (±0.0014), 70.4% for GPT-3.5 Turbo (±0.0013), 69.4% for Mistral 7B (±0.0017), and 69.3% for Falcon 7B (±0.0014). BERT SQuAD, included as a simpler benchmark, scored 50.83% (±0.0015), which the authors describe as not better than random choice. The authors report 95% confidence intervals based on 100 bootstrap iterations. For a separate party-platform rule based on participants’ self-reported political orientation, the models were more accurate in 21 of 24 comparisons, with exceptions for liberals choosing Bolsonaro-platform proposals for Mistral, Falcon, and Gemma. In one reported subset—left-identified participants choosing Lula-platform proposals—all six models reached at least 80% accuracy versus 71% for the rule. For choices between proposals from the same candidate’s program, reported model accuracy was 65%–77%. These comparisons use a specific rule and study setting, not a general-purpose benchmark.
What remains open
- Overall individual-choice accuracy figures for LLaMA-3 8B and Gemma 7B are not supplied.
- The supplied evidence does not give model-by-model exact results for every party-rule comparison or a single overall score against that rule.
Conditions and limits
- These are results reported by the authors for this participant group, set of proposals, and Brazilian 2022 election setting; they do not establish accuracy for other populations or settings.
- The party-platform rule is a specific comparator based on self-reported political orientation, not every possible alternative.
- The reported ± values are reproduced as given; the source describes 95% confidence intervals based on 100 bootstrap iterations.
- Accuracy varied across participant groups, and predictions could be sensitive to the order of options.
Objections to consider
- Accuracy varied with participants’ political orientation and education. This materially limits treating the reported overall figures as equally representative of every participant group; the supplied evidence does not explain the causes or establish how the differences generalize.
- The reported option-order sensitivity, including consistency below 70% in some cases, qualifies the stability of choice prediction. The supplied evidence does not quantify how often this occurred across all tested cases.