语言模型的价值判断和选择
Language models are frequently employed in settings where they are asked to make value judgments and choices. These observed choices often exhibit intransitivity: A model may prefer item $A$ to $B$ and $B$ to $C$, while also preferring $C$ to $A$.
Existing work that models LLM preferences treats such inconsistencies as sampling noise around a single latent ordering. We instead propose that intransitivity reflects the aggregation of multiple latent, internally consistent orderings.
模型的不一致性
We first show that observed inconsistencies cannot be explained by a single ordering under any monotone link function.
噪声增强混合模型
We then introduce a noise-augmented mixture Bradley-Terry (MBT) model that infers latent preference components from repeated pairwise comparisons.
模型解释力
Across seven models and four tasks, a mixture of orderings often explains structural inconsistencies better than single-utility models.
偏好异质性
We find that aggregate preferences often hide underlying preference heterogeneity.
道德机器案例分析
A case study on Moral Machine dilemmas shows that models which disagree on aggregate orderings can still share latent components.
多偏好反映
Together, these results suggest that LLMs reflect plural preferences.
对齐和评估流程
Alignment and evaluation pipelines that treat LLM preferences as a single function, therefore, risk averaging over coherent orderings that different users may endorse differently.