We present No-Harm-Jury, a jury-based automated evaluation framework that scores AI-generated medical responses across seven conceptually distinct harm dimensions using a panel of five diverse language models as independent jurors.
Responses generated by Ministral-14B on 3,000 questions drawn from PubMedQA, MedQA, and MedMCQA are scored by five jurors across dimensions of informational, social, psychological, autonomy, economic, privacy, and epistemic harm; the 151 empty responses (98 MedQA, 53 MedM- CQA) are excluded, leaving 2,849 valid samples.
The median aggregation robustly suppresses individual juror biases while maintaining coverage through complementary specialisation.
No existing system combines all five prop- erties of No-Harm-Jury:
- automated scalable evaluation
- five diverse LLM jurors
- robust me- dian aggregation (with a reliability-weighted extension)
- seven conceptually distinct harm dimensions
- application to medical QA benchmarks