Contribution to the handbook
Supporting · Assessed 2026-10-28
The question concerns a limited comparison measure that may be useful when evaluating models. The evidence says Kendall rank correlations were reported for six models on a test set containing 50% of participants, but supplies neither the values nor the specific ranking pairs. It therefore offers methodological context rather than a substantive design finding.
Does this challenge the goal?
In-scope gap
A model-comparison measure can inform testing within the accepted goal; asking about it does not challenge the goal or its assumptions.
The supplied evidence identifies Kendall rank correlation and the test-set scope, but gives no correlation values or specific ranking pairs.
Why this category?
- testing
- useful background
- evidence limitations
Without the reported values or a clear account of which rankings were compared, the packet does not support conclusions about the degree of model agreement.
This is an AI assessment of contribution to the goal, separate from whether the question has an answer.
Tests, stop conditions and unresolved work →
Read the exploration goalState whether rankings were compared and identify the reported comparison measure and scope, without inventing correlation results.
What we can say so far
Yes. The researchers report using Kendall rank correlations (Kendall’s tau) to compare preference rankings for six models on a test set containing 50% of participants. The supplied evidence does not give the correlation values or identify the specific ranking pairs.
What remains open
- The specific ranking pairs and their correlation values are not provided.
Conditions and limits
- This is an interpretation of the supplied source material, not independent verification.
- Figures were not independently interpreted; the supplied passage does not report correlation values.
Evidence and related ideas
Supporting evidence, qualifications and context play different roles in this answer.
context
Models in the comparison The comparison includes six language models and BERT SQuAD, a simpler benchmark. The models differ in what is known about their training: GPT 3.5 Turbo has proprietary architecture and data; LLaMA-2, LLaMA-3, Mistral, and Gemma are open-weight models trained on proprietary data; Falcon is open-weight and trained on public data; and BERT is a foundation model trained on public data.
Describes the comparison set as six language models plus a simpler BERT benchmark, providing context for interpreting the reported scope.
Large Language Models (LLMs) as Agents for Augmented Democracy
Read full source: Large Language Models (LLMs) as Agents for Augmented Democracy (opens in a new tab) ↗Read source discussion: Large Language Models (LLMs) as Agents for Augmented Democracy →
supports
Measures of agreement The authors use accuracy and report Kendall’s tau as another way to compare choices. They say F1 is not a distinct measure for these data because each correct choice creates both a true positive and a true negative, while each incorrect choice creates both a false negative and a false positive; as a result, precision, recall, F1, and accuracy are equal.
Identifies Kendall’s tau as a measure the authors use to compare choices, but does not provide results.
Large Language Models (LLMs) as Agents for Augmented Democracy
Read full source: Large Language Models (LLMs) as Agents for Augmented Democracy (opens in a new tab) ↗Read source discussion: Large Language Models (LLMs) as Agents for Augmented Democracy →
supports
Comparing preference rankings The researchers report Kendall rank correlations for six models on a test set containing 50% of participants, but the supplied passage gives no correlation values.
Directly reports Kendall rank correlations for six models on a test set containing 50% of participants, while noting that values and specific ranking pairs are not supplied.