A span tagger that ranks its outputs by risk can defer the riskiest ones to a person, and a second model can sharpen this ranking. Which second model should it be?
Partner Selection: Conditional on Primary's Errors
Partner selection is conditional, not global: the best second opinion is the model that is right where the primary is wrong.
Across eight tagger types, three seeds and three English benchmarks (576 evaluated primary-partner pairings), choosing the partner by held-out accuracy on the primary's errors recovers 82% of the best partner's gain, against 69-70% for span F1 or a random pick.
Frozen before a fourth dataset was chosen, the rule replicated there (Spearman 0.40 vs. 0.04 for span F1).
Partner Value: Directional and Distribution-Based
Partner value is also directional. The predictive distribution is the second opinion: hard labels discard half of a partner'