Contribution to the handbook
Priority · Assessed 2026-10-28
Group-level differences in prediction accuracy are a consequential limitation to investigate if models are used to represent citizens’ preferences. The evidence reports age- and model-specific sex patterns, but does not establish causes, effect sizes, or real-world political underrepresentation. The question therefore contributes to risk assessment and subgroup testing without proving harm.
Does this challenge the goal?
In-scope gap
Examining whether system performance varies among groups is within the definition’s existing citizen-service and testing goals. The question does not challenge those goals.
The supplied findings concern one study’s participants and models; they do not establish how differences would appear in other populations or governance settings.
Why this category?
- Identifies reported variation in accuracy across participant groups and models, relevant to assessing representation risks.
- Supports testing performance across groups rather than relying on aggregate accuracy alone.
- The reported directions and significance details have limits: the table summaries do not supply effect sizes, and broader applicability is unresolved.
The evidence is AI interpretation of one shared study source, not independent replication. In particular, table summaries identify significance patterns but do not give group averages or effect sizes; broader implications remain uncertain.
This is an AI assessment of contribution to the goal, separate from whether the question has an answer.
Authority, accountability and resilience →
Tests, stop conditions and unresolved work →
Read the exploration goalDescribe the age groups compared and which models showed age-related accuracy differences, including their direction.
What we can say so far
The study compared participants in the younger and older age quartiles. Mistral 7B, Falcon 7B, and Gemma 7B were reported to predict younger-quartile participants slightly more accurately, with statistically significant differences. The table also reports significant age-related differences for those three models, but not for GPT-3.5 Turbo, LLaMA-2 7B, or LLaMA-3 8B at its stated significance levels.
What remains open
- The supplied evidence does not give effect sizes or age-group averages for the model-specific comparisons.
Conditions and limits
- The comparison is between the younger and older quartiles, not all possible age groups.
- The reported direction and significance describe the study; the supplied evidence does not establish why the differences occurred or whether they would recur.
- The table summary gives significance but not effect sizes or age-group averages.
Objections to consider
- This table summary alone does not establish which age group was more accurately predicted; the supplied model-specific results provide that direction.
Evidence and related ideas
Supporting evidence, qualifications and context play different roles in this answer.
context
Comparing participant groups The authors plan to examine whether models predict choices differently across participants’ self-reported political ideology, age, education, and sex. For these comparisons, they balance the groups by downsampling the group with more participants; the supplied text does not include the results.
Describes the demographic-comparison method and notes that results are absent from that passage; it does not provide the age findings.
Large Language Models (LLMs) as Agents for Augmented Democracy
Read full source: Large Language Models (LLMs) as Agents for Augmented Democracy (opens in a new tab) ↗Read source discussion: Large Language Models (LLMs) as Agents for Augmented Democracy →
context
Uncertainty around accuracy figures The authors report 95% confidence intervals based on 100 bootstrap iterations for individual-model accuracy, and 99% confidence intervals based on 100 iterations for demographic comparisons.
Describe how the direction of sex-related accuracy differences varied by model.
What we can say so far
The reported direction differed by model: LLaMA-2 7B, Mistral 7B, and Falcon 7B were more accurate for female participants, whereas LLaMA-3 8B and Gemma 7B were more accurate for male participants. The supplied evidence does not state the direction for GPT-3.5 Turbo.
What remains open
- The supplied evidence says GPT-3.5 Turbo was not statistically significant at the table’s stated levels, but does not report whether its accuracy was higher for female or male participants.
Conditions and limits
- These are reported findings for the models and participants examined in the study.
- The evidence does not explain why the directions differed or whether the pattern would recur.
Objections to consider
- The broad statement that some models favored women does not establish a uniform direction. The detailed model-by-model result reports both female- and male-participant advantages.
- The significance table does not state GPT-3.5 Turbo’s direction, and the supplied directional results do not cover that model.
Evidence and related ideas
Supporting evidence, qualifications and context play different roles in this answer.
context
Comparing participant groups The authors plan to examine whether models predict choices differently across participants’ self-reported political ideology, age, education, and sex. For these comparisons, they balance the groups by downsampling the group with more participants; the supplied text does not include the results.
Describes demographic comparisons and balancing methods, but provides no sex-related accuracy directions.
Large Language Models (LLMs) as Agents for Augmented Democracy
Read full source: Large Language Models (LLMs) as Agents for Augmented Democracy (opens in a new tab) ↗Read source discussion: Large Language Models (LLMs) as Agents for Augmented Democracy →
context
Uncertainty around accuracy figures The authors report 95% confidence intervals based on 100 bootstrap iterations for individual-model accuracy, and 99% confidence intervals based on 100 iterations for demographic comparisons.