首页 > AI前沿 > A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks

A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks

arXiv机器学习 2026-07-02 21:46 7 阅读 查看原文

Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinical performance is far from solved - its "Hard" subset top score remains 32%.

We present a small, deliberately difficult evaluation dataset of five clinician-authored clinical scenarios spanning four specialties (anaesthesia, internal/family medicine, emergency medicine, and obstetrics), each accompanied by an atomic, weighted, MECE rubric (25-62 criteria per task; 184 criteria total) authored from a clinician-drafted golden answer.

We evaluate three frontier models: GPT 5.4, Claude Opus 4.7, and Gemini 3.1 Pro. Mean rubric pass rates were 0.47 (Claude), 0.38 (GPT), and 0.37 (Gemini).

The central finding is an inversion of clinical priority: the highest-weighted (weight-5, critical) criteria passed at only 32.4-41.7%, while low-stakes weight-1 criteria passed at 80-90%.

55 of 108 critical (weight-5) criteria (51%) were satisfied by no model.

Three LLM autoraters reproduced expert met/not-met labels on 92.8-94.6% of 552 graded criteria.

We position this as a methods-and-preliminary-findings contribution: the five tasks demonstrate a scalable, defensible pipeline ready to develop into a large-scale benchmark.