Contribution to the handbook
Priority · Assessed 2026-10-28
This supplies consequential empirical evidence about a proposed method for extending measured preference data: in the reported Type I B test, some sample matches improved when model predictions were added. The records also limit that result to aggregate preferences in the study group, note missing numerical detail for one tested sample size, and report concerns such as option-order sensitivity and group accuracy differences. Together these are material to assessing the method and its testing needs, not proof of real-world representation.
Does this challenge the goal?
In-scope gap
The question evaluates a proposed way to estimate group preferences, including its measured accuracy and limits, within the existing goals of citizen service and testing; it does not challenge those goals.
The reported improvement is on a study-specific measure using study data, not evidence that predicted preferences represent people outside the sample or improve real governance decisions.
Why this category?
- Empirical test of a proposed preference-estimation method
- Consequential limits and risks relevant to evaluation
The records come from a shared paper and therefore are not independent corroboration. Better alignment with the study sample’s aggregate measure does not establish accuracy for unstudied citizens, fairness across groups, public acceptance, or suitability for binding decisions.
This is an AI assessment of contribution to the goal, separate from whether the question has an answer.
- Did adding model predictions for more people improve the study's measure of sample accuracy?
- Ways to add preference data
- Testing whether a model improves a sample
- Testing group-level preferences
- The data used in the comparison
- Uncertainty around accuracy figures
- Choices and option order
- Some groups may be missing
- How the study measured a match
- Not all models or prompts were tested
- Type III A showed similar results
- Predictions improved some sample matches
- Small samples improved in this test
- Changing option order could change answers
- Summarizing group preferences with win rates
- Estimates of group preferences
- Comparing sample and full-sample patterns
- Different training-sample sizes
- Measures of agreement
Making and interpreting laws →
Authority, accountability and resilience →
Tests, stop conditions and unresolved work →
Read the exploration goalEstablish whether adding model predictions for more people improved the study’s measure of sample accuracy, with the relevant tested conditions and results.
What we can say so far
Yes. In the study’s Type I B test, adding model predictions improved the R² measure of how closely a sample’s aggregate preference pattern matched the full study sample, especially for smaller initial samples. With a 5% initial sample and predictions added for 20% of the people outside it, R² rose from about 30% to about 75%. With a 25% initial sample, the reported gain was about 7–10 percentage points. The supplied results give no separate numerical result for the 50% sample. These are results on the study’s participant data and tested models, not evidence that the method will generalize to other populations or settings.
What remains open
- No separate numerical result is supplied for the 50% initial sample.
- The supplied evidence does not establish performance on other datasets, populations, or real-world decisions.
- The supplied results do not quantify uncertainty around the reported augmentation gains.
Conditions and limits
- The measure is R² for matching sample-level proposal win-rate patterns to the full study sample; it does not establish that every individual’s preferences were reproduced or that the sample represents people outside the study.
- The quantified Type I B results used models fine-tuned on the same preference data and added predictions for 20% of people outside initial samples of 5%, 25%, and 50%.
- The supplied text reports the approximate R² increase for the 5% sample and a 7–10 percentage-point gain for the 25% sample, but no separate numerical result for the 50% sample.
- The finding is limited to the study data and reported tests; model coverage, prompt variation, and prediction consistency impose further limits.
Objections to consider
- The potential underrepresentation of groups missing from training data limits claims beyond the tested participant data; the supplied evidence does not measure or resolve that risk.
- The study did not test some newer models or many prompt variations, so the reported result cannot establish performance across those conditions.
- Option-order changes sometimes altered predictions, including reported cases below 70% consistency; the supplied evidence does not establish how this affects the augmentation gains overall.
- Consistency differed across models and exact model-by-model rates are not supplied, limiting claims of uniform reliability.
Evidence and related ideas
Supporting evidence, qualifications and context play different roles in this answer.