A starting point for discussion
The idea
The comparison includes six language models and the BERT SQuAD question-answering test, a simpler benchmark (a standard test for comparing models). The models differ in size and in what is known about their training: GPT 3.5 Turbo has an architecture and data that are not publicly available; LLaMA-2, LLaMA-3, Mistral, and Gemma are open-weight models, meaning their model weights are available, trained on data that are not publicly available; Falcon is open-weight and trained on public data; and BERT is a foundation model, a model trained on broad data that can be adapted for different tasks, trained on public data.
Place in the reasoning
AI assessment · Connected to reasoning
Distinguishes models by access to weights and training data. Those development differences bear on control and transparency concerns for personal agents, but do not demonstrate how any proposed civic agent would be governed.
Matching views does not make AI representation accountable
The distinction between open or proprietary weights and training data is relevant to model control and transparency in personal agents; it does not establish the governance or accountability of a proposed civic agent.
What this means and why it matters
What this means
The study compares models of different sizes and training backgrounds, including a simpler test for comparing models. Open-weight means the model’s learned values, called weights, are available; it does not necessarily mean the training data were public.
Why it matters
These differences matter when interpreting what the comparison can show. It covers the models the authors could fine-tune at that time, not every available model.
Conditions & assumptions
Source coverage: Full readable text of the retrieved 37-page research draft, including references and appendices. Figures are not independently interpreted. The authors include a simpler benchmark to examine whether a larger model is needed for this task. They also report differences in models’ access to weights and training data. GPT-4 and Claude 3 were not included because options to train them further for a particular task were not publicly available to the authors at the time.
Questions this connects to
AI-suggested relationships. Authors have not necessarily responded to one another.