A starting point for discussion
The idea
The researchers defined a consistency rate as the share of choices between two options that stayed the same when the order of the two options was reversed. They reported rates across different models and sizes of samples selected using chance. GPT-3.5 Turbo and Falcon 7B were generally consistent in more than 70% of cases; the other models did not show this level of consistency despite similar accuracy. The excerpt gives no exact rates for each model.
Place in the reasoning
AI assessment · Connected to reasoning
Reports option-order consistency for particular models and samples, including that similar accuracy did not imply similar consistency; it does not give exact rates for each model.
Different prediction checks reveal different limits
Directly illustrates that accuracy does not alone establish reliability under reversed presentation; findings remain limited to the tested models and samples.
What this means and why it matters
What this means
A model counted as consistent when it chose the same proposal whether that proposal appeared first or second. The researchers compared this measure across models and sample sizes.
Why it matters
Accuracy alone may not reveal whether a model's choice depends on how options are presented. The reported excerpt does not give exact rates for each model or sample size.
Conditions & assumptions
Source coverage: Full readable text of the retrieved 37-page research draft shared before formal publication, including references and appendices. Figures are not independently interpreted. Passages 42–43 define the measure and report that results were presented across models and sizes of probabilistic samples. They give a broad comparison but not exact rates for each model. Addresses: Consistency under option-order changes
Questions this connects to
AI-suggested relationships. Authors have not necessarily responded to one another.