Contribution to the handbook
Priority · Assessed 2026-10-28
The question supplies a concrete validation method and consequential qualification: stable sample estimates or a close match to the study’s full sample do not establish representativeness beyond it. This is central to assessing whether citizen-preference estimates could responsibly inform a system.
Does this challenge the goal?
In-scope gap
The question investigates the evidence and limits of sample adequacy for preference estimates, within the existing citizen-serving objective; it does not challenge that objective.
The study’s checks address consistency within its participant data, not whether those participants represent a wider public.
Why this category?
- testing
- consequential risks and limits
The findings and method come from one study and measure stability or fit within that study’s data. They do not demonstrate that the estimates reflect citizens outside the sample.
This is an AI assessment of contribution to the goal, separate from whether the question has an answer.
- How did the researchers check whether they had enough participant data?
- A check on preference stability
- Group preferences were consistent on retest
- The data used in the comparison
- How the study measured a match
- Summarizing group preferences with win rates
- Reported stability in the sample
- Comparing sample and full-sample patterns
Authority, accountability and resilience →
Tests, stop conditions and unresolved work →
Read the exploration goalDescribe how the researchers compared samples and measured the stability of preference estimates as sample size varied.
What we can say so far
The researchers assessed stability by repeatedly comparing equal-sized, non-overlapping random participant samples. For each sample, they calculated each proposal’s win rate, then compared the win-rate patterns using the coefficient of determination (R²) for the Pearson correlation. They plotted this similarity against sample size. They reported that two samples of 53 participants each—about 20% of the study sample—would have an R² of about 0.72, and that the full-sample test-retest comparison had an R² of 0.9538. These results describe consistency within the study sample, not representativeness of the wider public.
What remains open
- The supplied evidence does not provide an explicit adequacy cutoff or decision rule beyond reporting stability estimates at sample sizes.
Conditions and limits
- The reported procedure and values are attributed to the researchers’ preprint and are not independently verified here.
- The comparison concerns stability of proposal win-rate patterns within the study data; it does not establish that the sample represents the wider public.
- The supplied excerpts do not state an explicit threshold for deciding that a sample size is adequate.
Objections to consider
- A match to the study’s full-sample aggregate preferences cannot establish representativeness beyond the participants studied. The answer confines its claim to within-study stability and states this limitation.
Evidence and related ideas
Supporting evidence, qualifications and context play different roles in this answer.
supports
Group preferences were consistent on retest The authors estimate 95.38% test-retest reliability for the group’s preferences in the full participant sample. They say this is consistent with earlier paired-comparison work, where rankings tended to converge after about 30 to 40 choices per participant.
Adds the authors’ full-sample test-retest reliability estimate and context, while noting that retest details are not provided.
Large Language Models (LLMs) as Agents for Augmented Democracy
Read full source: Large Language Models (LLMs) as Agents for Augmented Democracy (opens in a new tab) ↗