首页 > AI前沿 > No-Harm-Jury A Jury-Based Multi-Dimensional Harm Evaluation Framework for Medical AI Responses

No-Harm-Jury A Jury-Based Multi-Dimensional Harm Evaluation Framework for Medical AI Responses

arXiv自然语言 2026-11-20 13:00 25 阅读 查看原文

We present No-Harm-Jury, a jury-based automated evaluation framework that scores AI-generated medical responses across seven conceptually distinct harm dimensions using a panel of five diverse language models as independent jurors.

Responses generated by Ministral-14B on 3,000 questions drawn from PubMedQA, MedQA, and MedMCQA are scored by five jurors across dimensions of informational, social, psychological, autonomy, economic, privacy, and epistemic harm; the 151 empty responses (98 MedQA, 53 MedM- CQA) are excluded, leaving 2,849 valid samples.

The median aggregation robustly suppresses individual juror biases while maintaining coverage through complementary specialisation.

No existing system combines all five prop- erties of No-Harm-Jury:

  • automated scalable evaluation
  • five diverse LLM jurors
  • robust me- dian aggregation (with a reliability-weighted extension)
  • seven conceptually distinct harm dimensions
  • application to medical QA benchmarks