Opening Nahir…Loading this idea…
← Explore questionsContribution to the handbook
Priority · Assessed 2026-10-28
Subgroup accuracy is material to whether a system using model predictions could represent citizens’ views reliably. The evidence reports differences across groups and models, while also limiting what those results establish. This makes the question relevant to representation safeguards and validation, even though it does not demonstrate unequal influence in an actual governance process.
Does this challenge the goal?
In-scope gap
The question examines subgroup accuracy within the existing goal of assessing whether governance arrangements serve citizens; it does not contest that goal.
The evidence is reported through AI interpretations of one shared source, and the results are limited to particular models, participants, and tests.
Why this category?
- material system risk
- citizen representation
- testing and limits
The reported differences do not establish their causes, persistence in other populations, or effects on representation in a real decision process.
This is an AI assessment of contribution to the goal, separate from whether the question has an answer.
Making and interpreting laws →
Authority, accountability and resilience →
Tests, stop conditions and unresolved work →
Read the exploration goalAssess reported success at reproducing group response patterns and variation in accuracy across groups, with limits on generalizing across models, questions, and settings.
What we can say so far
The supplied evidence indicates that accuracy differed across some participant groups in the study, but it does not establish a single overall level of success at reproducing group response patterns. The authors report higher accuracy for liberal than conservative or centrist participants, and for college-educated than non-college-educated participants. Age and sex patterns were model-dependent: some models were reported as more accurate for younger participants, while the direction of sex-related differences varied across models. Earlier studies cited in the evidence also reported subgroup reproduction, including small gender-related differences, but those reports are not fully detailed here. These findings support a qualified conclusion: performance was not uniform across groups, and the available evidence does not justify generalizing the results to other models, questions, settings, or populations.
What remains open
- The supplied evidence does not give an overall quantified success level for reproducing group response patterns.
- For the principal reported group differences, the supplied passages do not provide group accuracy averages or effect sizes sufficient to judge their practical magnitude.
- The evidence does not establish whether the reported differences recur across other questions, settings, or populations, or explain their causes.
- The evidence does not measure whether accuracy differences result in actual overrepresentation or underrepresentation in a decision process.
Conditions and limits
- The reported study results concern its participants and tested models; the evidence does not establish that they generalize to other populations or language models.
- The age comparison concerns younger and older quartiles. The direction of reported age and sex differences varied across models.
- The reported significance tests do not by themselves establish practical importance or the size of group differences.
- Findings attributed to earlier studies are prior-study reports, not new results from the study described here, and the supplied passages provide limited detail for evaluating them.
- The study compares proposal win-rate patterns with its full participant sample; that match is not equivalent to reproducing every individual’s preferences or representing a wider public.
- The evidence is based on interpretations of the same retrieved preprint, so repeated passages do not constitute independent corroboration.
Objections to consider
- The significance table does not give the direction or magnitude of age-group differences, so it cannot establish how practically important those differences are.
The table reports significant sex-group differences for several models but does not provide their size; the supplied direction information does not resolve practical magnitude.The limited invariance findings do not establish equal accuracy across groups; the reported accuracy differences elsewhere in the study support retaining a qualified, non-uniform conclusion.The concern about groups missing from training data is not quantified, and the supplied evidence does not identify which groups are affected or the size of any resulting fidelity gap.Observed accuracy gaps raise a representation concern, but the evidence does not measure actual overrepresentation or underrepresentation in a decision process.Reported option-order sensitivity may limit reliability, but the passage does not establish how prevalent it was across all tested cases or how it affected subgroup fidelity.The described statistical tests identify a method for comparing group averages but do not supply the averages or effect sizes needed to judge practical importance.The study's participant scope is explicitly limited, so the answer confines its conclusions to the tested participants and does not generalize them to Brazil's population.The answer describes subgroup accuracy variation but does not equate matching proposal win-rate patterns with reproducing every individual preference or representing a wider public.Evidence and related ideas
Supporting evidence, qualifications and context play different roles in this answer.
supports
Age and sex results varied
Age comparisons used participants in the younger and older quartiles. Mistral 7B, Falcon 7B, and Gemma 7B showed a slight, statistically significant tendency to predict younger-quartile participants more accurately. Sex-related accuracy varied by model: LLaMA-2 7B, Mistral 7B, and Falcon 7B were more accurate for female participants, while LLaMA-3 8B and Gemma 7B were more accurate for male participants.
Reports age- and sex-related accuracy directions and shows that the sex pattern differs by model; supports the qualified subgroup-variation conclusion.
qualifies
Age-group accuracy results
For younger-versus-older participants, the table reports significant accuracy differences for Mistral 7B, Falcon 7B, and Gemma 7B, each marked *** for the 1% significance level. GPT-3.5 Turbo, LLaMA-2 7B, and LLaMA-3 8B are not marked significant at the table’s stated levels.
Reports which models had significant younger-versus-older accuracy differences, but gives neither the direction nor size of those differences.
qualifies
Sex-group accuracy results
For female-versus-male participants, the table reports significant accuracy differences for LLaMA-2 7B, Mistral 7B, Falcon 7B, LLaMA-3 8B, and Gemma 7B. Each is marked *** for the 1% significance level. GPT-3.5 Turbo is not marked significant at the table’s stated levels.
Reports significant sex-group accuracy differences for several models, but does not give their direction or size.
supports
Findings on subgroup predictions
A study by Argyle and colleagues found that language models could reproduce response patterns across many human subgroups. In their comparison by gender, the models were about one percentage point more accurate at predicting women’s preferences.
Attributes to an earlier study reports of reproducing response patterns across subgroups and a roughly one-percentage-point gender difference; its limited detail and transferability qualify its use.
qualifies
Results changed little in tested variations
The authors report that accuracy did not depend strongly on which demographic details were used. The tested categories were political ideology (liberal, conservative or centrist), location (city and state), sex (male or female) and education (non-college educated or college educated). The demographic tests used GPT-3.5 Turbo and Mistral 7B. They also report little change when prompts and consultation switched from English to Portuguese.
Reports little change under specified demographic-input and language variations for two models, while explicitly limiting the result to those tests; it does not establish equal accuracy across groups.
supports
Accuracy varied across groups
A study by Santurkar and colleagues reported that its models more accurately reproduced preferences among young, low-income, politically moderate people with less than high-school education. It also found slightly higher accuracy for men than women, by about one percentage point or less.
Attributes to an earlier study reported accuracy differences among demographic groups, including a small gender difference; the supplied passage does not provide its methods or broader stability.
qualifies
How the study measured a match
The study compares each proposal's win rate in a sample with its win rate in the full participant data, then uses R² to measure how closely the sample reproduces the full-data pattern.
Clarifies that the study's accuracy measure compares a sample with the full study sample, rather than directly testing representation of people outside the study.
context
Summarizing group preferences with win rates
To estimate group-wide policy preferences, the authors define each proposal’s win rate as the share of recorded pairwise choices in which it was selected, then compare win rates from a sample with those from the full study sample.
Describes win rates as the group-preference summary and limits the comparison to the study's full sample.
qualifies
Some groups may be missing
Biases and other limits may make language models worse at representing some political views, especially views of groups missing from training data.
Records the authors' concern that groups poorly represented in training data may be represented less well, while noting that the passage does not measure which groups or the size of any effect.
context
Models can imitate some responses
Earlier studies cited by the authors found that language models could simulate some opinion-poll responses and predict votes in elections with two choices.
Reports earlier work on simulated opinion-poll responses and two-choice vote prediction, without evidence about performance across groups.
qualifies
Accuracy gaps raise a representation concern
The authors connect the accuracy gaps across participant groups to the wider question of whether language models may represent some parts of the public more than others.
Frames group accuracy differences as a representation concern but does not establish actual overrepresentation in a decision process.
supports
Differences by political view and education
Across the six models, the authors report higher accuracy for liberal participants than for conservative and centrist participants, and higher accuracy for college-educated than for non-college-educated participants.
Reports higher accuracy for liberal than conservative and centrist participants, and for college-educated than non-college-educated participants; directly supports observed subgroup variation.
context
Earlier studies simulated some views
Earlier studies cited by the authors reported similar moral judgments from language models and used detailed personas to predict election choices.
Summarizes earlier work on moral judgments and election choices but supplies no group-specific performance estimates.
qualifies
Changing option order could change answers
The authors report that model predictions could be unclear and could change when the order of two options was reversed. In some cases, consistency was below 70%.
Reports that predictions could change with option order and that consistency was below 70% in some cases, a limitation on interpreting prediction reliability.
context
Six fine-tuned models
The authors fine-tune six existing language models on the volunteer data: Llama-2 7B, LLaMA-3 8B, ChatGPT 3.5 Turbo, Mistral 7B, Falcon 7B, and Gemma 7B.
Identifies the six fine-tuned models, helping bound the setting but not establishing performance beyond it.
qualifies
Choices and option order
The researchers defined a consistency rate as the share of pairwise choices that stayed the same when the order of the two options was reversed. They reported rates across different models and sizes of probabilistic samples. GPT-3.5 Turbo and Falcon 7B were generally consistent in more than 70% of cases; the other models did not show this level of consistency despite similar accuracy. The excerpt gives no exact rates for each model.
Reports model variation in consistency under option-order changes, with exact model-specific rates unavailable; qualifies claims of uniform reliability.
qualifies
Testing group accuracy
To compare average accuracy between participant groups, the researchers used two-sample t-tests that allow the groups to have unequal variances. The tests used models trained on half of the participant data.
Describes the group-comparison tests and their training scope, while noting that the tests alone do not provide effect size or practical importance.
context
Unequal data access is a concern
The authors say richer digital representations could use more or better data, but access to such data is uneven across the population.
Notes uneven access to data as a possible challenge, but says the study does not measure its effect on representation.
context
People were represented with limited data
The study represented each person using a short set of demographic details and pairwise preferences. The authors say this is a limited form of a digital twin, meaning a digital representation of a person.
Describes the limited demographic and preference information used to represent each person; it does not test whether richer information improves fidelity.
qualifies
More limitations may remain
The authors say the limitations they list are only some of the many challenges their approach may face.
States that the authors' listed limitations are non-exhaustive, reinforcing that the evidence should not be treated as a complete account of limits.
qualifies
Bias may differ across tasks
The authors distinguish bias in language tasks, such as assumptions linking jobs with gender, from bias in predicting civic preferences. They caution that a bias found in one task should not simply be assumed to operate in the other.
Warns against assuming that biases observed in other language tasks transfer to civic-preference prediction without evidence.
context
The study tests preference models
The authors say the study explores different forms of augmentation and develops benchmarks for individual and group preferences.
States the study's aims and benchmarks but does not itself establish their success.
qualifies
The study’s scope
The study asks whether models can learn preferences within a specific group of 267 participants, not estimate what Brazil’s population prefers.
Limits the study target to 267 participants and says the data cannot support estimates for Brazil's general population.
qualifies
Comparing sample and full-sample patterns
The authors compare sample and full-sample win rates using the squared Pearson correlation, treating a value of 100% as identical win-rate patterns.
Clarifies that squared correlation measures similarity in proposal win-rate patterns, not fidelity to individual choices or representation of a wider public.