A starting point for discussion
The idea
In a test trained on 50% of participants and evaluated on choices between two options from the other 50%, LLaMA-2 had 76.68% accuracy (reported uncertainty: ±0.0014), GPT 3.5 Turbo 70.4% (±0.0013), Mistral 7B 69.4% (±0.0017), Falcon 7B 69.3% (±0.0014), and BERT SQuAD 50.83% (±0.0015). The authors describe BERT SQuAD’s result as no better than random choice.
Place in the reasoning
AI assessment · Connected to reasoning
Reports model-specific accuracy on held-out study participants’ pairwise choices, with BERT described as no better than random; this is evidence about a particular test, not civic-system performance.
Different prediction checks reveal different limits
The reported accuracy figures and test setup directly concern prediction quality, while remaining limited to these models, participants, and data.
Matching views does not make AI representation accountable
Prediction accuracy on held-out recorded choices does not demonstrate that personalized agents can represent people accountably in real decisions.
What this means and why it matters
What this means
The figures show how often each model matched participants’ recorded choices between two options in this test of 267 participants. The values after ± show the reported uncertainty amounts.
Why it matters
The results compare models on this dataset and test design. They do not establish prediction accuracy for Brazil’s wider population or for real decisions.
Conditions & assumptions
Source coverage: Full readable text of the retrieved 37-page research draft, including references and appendices. Figures are not independently interpreted. The authors report these accuracy figures for a test set made up of half the participants, after training on the other half. They estimate 95% confidence intervals, ranges used to show uncertainty around a result, with 100 bootstrap iterations. Bootstrapping means repeatedly resampling the data.
Questions this connects to
AI-suggested relationships. Authors have not necessarily responded to one another.