What should readers know about the human comparison?

Contribution to the handbook

Supporting · Assessed 2026-10-27

The assignment, training, incentive, and procedural differences help readers interpret a reported AI-versus-human comparison without treating it as a definitive general ranking. This is useful qualification for the tests section, but is study-specific rather than a governance design in itself.

Does this challenge the goal?

In-scope gap

The question seeks qualifications for interpreting a comparison relevant to testing; it does not contest the handbook’s goal or assumptions.

The evidence reports procedural differences but does not establish their effects on the comparison.

Tests, stop conditions and unresolved work →

Identify how human mediators were assigned, trained and incentivized, and note procedural differences that matter when interpreting the comparison.

What we can say so far

In the reported comparison, one person in each six-person group was randomly assigned to mediate. Human mediators received training and were offered a financial bonus if their statement was the group’s most preferred; the available account does not describe the training content. The human-mediator task did not include the AI helper’s critique phase. It instead used four rounds, with outside raters assessing transcript quality. The authors characterized writing a group-preferred statement as challenging for human mediators. These procedural differences matter when interpreting the comparison: the human and AI processes were not identical.

What remains open

• The available evidence says mediators received training but does not describe its content.