首页 > AI前沿 > Same Judgement, Different Scores: Numerical Scoring Topology in LLM-as-a-Judge Evaluation

Same Judgement, Different Scores: Numerical Scoring Topology in LLM-as-a-Judge Evaluation

arXiv自然语言 2026-11-19 21:17 12 阅读 查看原文

研究背景与问题

LLMs are increasingly used as automated evaluators to reduce the cost and time required for human annotation. However, the role of scoring-scheme design in the reliability of LLM-as-a-Judge evaluation remains underexplored.

研究方法与设计

To address this problem, we introduce numerical scoring topology, which characterizes the structural design of a scoring scheme in terms of its scale width, granularity, offset, symmetry, and boundary placement. Building on this definition, we conduct the first systematic study of how numerical scoring topology influences zero-shot LLM-as-a-Judge evaluation across diverse benchmarks and judge models.

研究结果与发现

Our results show that numerical scoring topology systematically affects judge scores, calibration, and robustness, particularly in judgment-intensive evaluations. Shifted and narrow ranges tend to induce some of the largest score deviations across many settings.

We further find that stronger and larger judge models generally achieve higher discriminative validity and greater robustness to scoring-topology changes.

结论与建议

These findings suggest that numerical scoring topology should be treated as an integral component of LLM-as-a-Judge evaluation and highlight the need for more principled and standardized scoring protocols.