The Prediction Question
A negotiation assistant can reach a good deal without accurately predicting the other person's response. We ask whether evaluations of social world models test that prediction directly.
Audit Methodology
We audit 113 works published from 1999 to 2026.
Primary Evaluations
- 9 primary evaluations measure a real person's response to an action selected by the system or experimenter.
- Only 2 also compare an explicit response forecast with that response.
Their predictors use information available when the action is chosen, and the evaluated responses were excluded from fitting; both randomize the intervention.
Language-Model Works
A separate full-text check of 25 language-model works finds no experiment meeting these conditions.
Alternative Evaluation Targets
The other evaluations test current-state inference, task success, or prediction from recorded actions.
Proposed Experiment
We propose a 1-step negotiation experiment that separates these questions.
Scope Limitation
The findings describe the audited sample, not the prevalence of this evaluation gap across the field.