A starting point for discussion
The idea
The authors say their exploration was limited: it did not include some newer models, such as GPT-4 and Claude 3, or test many versions of the instructions given to the models during training and use.
Place in the reasoning
AI assessment · Connected to reasoning
The source records limited model and prompt coverage, without claiming that untested models or prompts would perform better.
Different prediction checks reveal different limits
The limited set of models and prompt variations narrows the scope of prediction-quality conclusions.
What this means and why it matters
What this means
The researchers tested a limited set of models and instructions, rather than comparing a broad range of current options.
Why it matters
The results may change with other models or instructions, so they do not settle which approach works best.
Conditions & assumptions
Source coverage: Full readable text of the 37-page research draft, including references and appendices. Figures are not independently interpreted. These are stated limits of the authors’ exploration; the passage does not claim that newer models or different instructions would perform better. Addresses: Limits of model and instruction coverage
Questions this connects to
AI-suggested relationships. Authors have not necessarily responded to one another.
How strong is this claim?
AI assessment · not a probability or a final resolution
The passage clearly identifies missing tests, while leaving their possible effect unknown.
See the assessment in detail
The authors explicitly list the models and different versions of the instructions given to them that they did not examine.