One approach to accelerating progress on AI alignment is to automate parts of alignment research itself. Doing so requires systems that can perform reliably on hard-to-supervise research tasks, including:
- identifying flaws in research proposals
- evaluating whether a proposed experiment supports its intended claim.
To evaluate this capability, we introduce Tampered ML, a benchmark of 1,858 item-pairs, each consisting of:
- an experiment proposal reconstructed from a published machine learning paper
- a minimally edited counterfactual that introduces a targeted methodological flaw.
Models must identify the more methodologically sound proposal in both presentation orders.
A human validation study on a stratified sample estimates that 86% of items are valid (95% CI: 81–91%).
Frontier models perform above chance but remain far from reliable (with Opus 5 achieving 77.9% accuracy). They are highly sensitive to the order of presentation and favor flawed designs that appear more specific, transparent, or conventional.