首页 > AI前沿 > Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models

Bradley-Terry Training Can Distort Outcome Probabilities of Initially Accurate Language World Models

arXiv机器学习 2026-11-19 14:42 14 阅读 查看原文

Comparative feedback, in which a judge says which of two candidate outcomes is closer to what happened, is a practical way to train language world models that assign probabilities to an action's outcomes.

We show that, fitted with a Bradley-Terry objective, it need not teach the true outcome probabilities and can distort accurate ones.

On a simulated robot push with enumerated outcome distributions, this objective takes a model that is accurate on unseen layouts (hazard error 2.4 percentage points, pp) and moves 33 to 43 pp of its probability onto outcomes that none of