A starting point for discussion
The idea
The authors report that accuracy did not depend strongly on which demographic details were used. The tested categories were political ideology (liberal, conservative or centrist), location (city and state), sex (male or female) and education (non-college educated or college educated). The demographic tests used GPT-3.5 Turbo and Mistral 7B. They also report little change when prompts and consultation switched from English to Portuguese.
Place in the reasoning
AI assessment · Connected to reasoning
Reports limited tests of sensitivity to demographic inputs and language, qualifying inferences from model-performance results without establishing robustness generally.
Different prediction checks reveal different limits
The limited variations tested do not establish general reliability across all demographic characteristics, models, or languages.
Good overall predictions can hide worse results for some groups
Reported stability across selected inputs should not be read as showing equal accuracy or inclusion for all groups.
What this means and why it matters
What this means
The authors changed the listed demographic details in tests with GPT-3.5 Turbo and Mistral 7B. They also compared prompts and consultation in English and Portuguese, and report little change in accuracy.
Why it matters
These results suggest that accuracy did not depend strongly on those particular choices. They do not show equal performance for other groups, languages or settings.
Conditions & assumptions
Source coverage: Full readable text of the 37-page research draft, including references and appendices. Figures are not independently interpreted. The report concerns particular demographic categories, two models and a change from English to Portuguese. It does not show that the method works equally well for all groups or languages. Addresses: Reported tests of changes in model inputs
Questions this connects to
AI-suggested relationships. Authors have not necessarily responded to one another.