Contribution to the handbook
Priority · Assessed 2026-10-28
The reported accuracy differences across several tested groups raise a consequential representation concern, and the language and demographic checks are directly relevant to evaluating a civic system. The evidence does not establish real-world exclusion or broader population effects.
Does this challenge the goal?
In-scope gap
Testing whether prediction results vary across demographic details and language adds evidence relevant to whether a system could serve citizens fairly; it does not contest the handbook’s goal.
The findings concern specified models, participant categories and English-to-Portuguese tests, not all groups or languages.
Why this category?
- consequential representation risk
- testing and evidence limits
The packet reports significance patterns and some directions, but does not provide effect sizes for several comparisons or establish causes, practical impact, or transfer to other populations.
This is an AI assessment of contribution to the goal, separate from whether the question has an answer.
- How sensitive were the reported results to the tested demographic details and language?
- Results changed little in tested variations
- Sex-group accuracy results
- Accuracy varied across groups
- Age and sex results varied
- Education and ideology differences
- Differences by political view and education
- Age-group accuracy results
Authority, accountability and resilience →
Tests, stop conditions and unresolved work →
Read the exploration goalDescribe how results varied with the demographic details and language tested, without extending the finding to untested groups or languages.
What we can say so far
The authors report little change when they varied the tested demographic details—political ideology, location (city and state), sex, and education—and little change when prompts and consultation switched from English to Portuguese. The reported robustness check used GPT-3.5 Turbo and Mistral 7B, so this finding should not be generalized to other models, groups, or languages. Age appears among the participant details used as prompt context, but the supplied evidence does not clearly report an age-specific result for this sensitivity check. Separately, the study reports differences in prediction accuracy across participant groups; those results concern accuracy for groups, not whether changing demographic details in prompts changed the results.
What remains open
- The supplied evidence does not quantify how much results changed for each demographic detail or clearly report the sensitivity-check result for age.
- No finding is established for untested groups, models, or languages.
Conditions and limits
- The reported robustness finding is limited to GPT-3.5 Turbo and Mistral 7B, the specified demographic details, and English-to-Portuguese changes.
- Age is listed among the demographic context used in prompts, but the supplied evidence does not clearly identify an age-specific sensitivity result.
- Group-level differences in prediction accuracy are distinct from sensitivity to changing demographic details in prompts.
- These are reported findings from the study, not independently verified results.
Objections to consider
- The reported accuracy differences across ideology and education could appear to conflict with a broad claim of demographic robustness. The answer limits the robustness claim to changing tested prompt details and distinguishes it from group-level prediction accuracy.
- The reported age- and sex-group accuracy patterns qualify any interpretation that robustness means equal accuracy across demographic groups. The answer distinguishes these group comparisons from prompt sensitivity.
- This reports group-level accuracy variation, which could be mistaken for a contradiction unless kept distinct from the tested prompt-detail variations.
- This reports group accuracy differences and model-specific tendencies, qualifying any broader interpretation of robustness; the answer limits its claim to the prompt-variation tests.
- Significant education and ideology group accuracy differences qualify any claim of equal accuracy, but do not directly test sensitivity to changing prompt details.
- The age-group significance results concern prediction accuracy across groups and do not provide an age-specific prompt-sensitivity result; the answer preserves that distinction and identifies the age-specific gap.