首页 > AI前沿 > Metric-Dependent Annotation Saturation for Learning from Label Distributions

Metric-Dependent Annotation Saturation for Learning from Label Distributions

arXiv自然语言 2026-05-28 19:46 3 阅读 查看原文
The user wants me to format the provided plain-text news/article text into a semantic HTML fragment, following strict rules. Let's analyze the text and plan the HTML structure: 1. First Sentence: "When annotators disagree on a label, the disagreement itself carries signal - and the number of annotators needed to capture it depends on the evaluation metric." - Action: This is a strong thesis statement, so I'll wrap it in a `

` tag and use `` for the whole sentence or key phrases. Actually, the rule says "重要句子可用". Let's make the first sentence bold. 2. Second Sentence: "We fine-tune NLI models on label distributions subsampled from ChaosNLI, a dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation." - Action: Wrap in `

`. I can highlight "ChaosNLI" and "metric-dependent saturation" with `` for better readability. 3. Third Sentence: "In our 3-class NLI setting, entropy correlation - whether the model identifies which items elicit disagreement - requires N ~ 20-50 annotators to converge, while distributional match (KL divergence) saturates by N ~ 10 (87-95% of improvement across five model seeds); the specific thresholds are estimates for this setting, and it is the phenomenon, not the constants, that we expect to generalize." - Action: This is a longer sentence detailing specific findings. I'll split it or keep it in one `

`. Let's keep it in one `

` for strict adherence to "保留原文所有信息,不得增删改内容" (don't add/delete/change content, including sentence boundaries). I will bold the key terms: "entropy correlation", "distributional match (KL divergence)", and "the phenomenon, not the constants". 4. Fourth Sentence: "This finding rests on a prior observation: soft labels carry item-specific signal that label smoothing cannot replicate." - Action: Wrap in `

`. Bold "soft labels carry item-specific signal that label smoothing cannot replicate". 5. Fifth Sentence: "Across five smoothing intensities, entropy correlation clusters at r ~ 0.45-0.49, while soft labels reach r = 0.643 (p < 0.001); per-item analysis traces this gap to smoothing's inability to distinguish ambiguous items from clear ones." - Action: This sentence contains statistical data. I could split it into two `

` tags or use a list, but since it's not a list, I'll keep it in one `

` or split at the semicolon. Let's use one `

` but maybe break it down with `` tags. Actually, the semicolon separates two distinct ideas. Let's keep it as one `

` to avoid altering sentence structure, but maybe I can use `` for the numbers. Or, I can just leave it as a standard paragraph. Let's use `

` and bold "r ~ 0.45-0.49" and "r = 0.643". Wait, "per-item analysis traces this gap to smoothing's inability to distinguish ambiguous items from clear ones" is a significant finding. 6. Sixth Sentence: "The soft label advantage replicates across two architectures (DeBERTa, RoBERTa) and a non-NLI-pretrained baseline; differential saturation additionally appears in an exploratory cross-domain evaluation on content safety." - Action: Wrap in `

`. Bold "The soft label advantage" and "differential saturation". 7. Seventh Sentence: "These results suggest that annotation budgets should be informed by the target evaluation metric rather than set uniformly." - Action: Wrap in `

` and make it the concluding statement. Bold the whole sentence or key parts. Let's refine the HTML structure. The prompt asks for: - `

` for paragraphs. - `

` or `

` for sections if it's a long text. This text is quite short (one paragraph in reality), but I can split it into sections to demonstrate the use of headings, or just use paragraphs. The rule says "长文用

/

划分小节". Since this is a short abstract, I'll just use `

` tags, or maybe one `

` for the main topic and then paragraphs. Let's just use paragraphs to be safe, or maybe add a heading like `

Annotator Disagreement and Evaluation Metrics

`. No, the rule says "长文用", implying if it's long. I'll just use `

`. - Wait, I can group the sentences into logical paragraphs. The original text is one single block. I should probably preserve the original paragraph structure. The original text is just one block of text. I will output it as a single `

` or split it into logical paragraphs if it improves semantics without changing content. Since the prompt says "每个自然段落用

包裹", and the input is one natural paragraph, I should probably wrap the whole thing in one `

`, or break it into multiple `

`s if there are clear semantic breaks. Let's break it into 3 paragraphs for better readability: - Para 1: The setup and main finding (sentences 1-3). - Para 2: The prior observation and the data (sentences 4-5). - Para 3: The conclusion (sentences 6-7). Actually, keeping it as one paragraph is safer to avoid "增删改内容" (adding/deleting/modifying content). Let's use one `

` for the whole text, or just format the sentences within it. But HTML doesn't allow `

` inside `

`. I can use `
` but that's not semantic. Let's split into logical paragraphs. The original text is one paragraph. I will just wrap it in one `

` and use `` for emphasis. Wait, "每个自然段落用

包裹" means each natural paragraph should be wrapped in `

`. If the input has only one natural paragraph, I output one `

`. Let's re-read carefully: "When annotators disagree... rather than set uniformly." This is one paragraph. I will wrap it in `

`. Wait, I can use `

` or `

` to make it look more like a structured article. Let's add a heading. `

Metric-Dependent Saturation in Annotator Agreement

` Then the paragraph. Let's check the rules again: 1. Only output HTML fragment, no markdown fences, no explanation. 2. Each natural paragraph wrapped in `

`, long text uses `

`/`

` for sections. 3. Lists with `
  • `, quotes with `
    `, code with `
    `, important sentences with ``.
    4. Preserve all original information, no adding/deleting/changing.
    5. No ``, ``, ``.
    
    Since the text is relatively short but dense, I can use `

    ` to introduce the topic, then break the text into 2 or 3 `

    `s for readability, ensuring no words are lost. Let's divide the text into 3 semantic paragraphs: Para 1: "When annotators disagree on a label, the disagreement itself carries signal - and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI, a dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation. In our 3-class NLI setting, entropy correlation - whether the model identifies which items elicit disagreement - requires N ~ 20-50 annotators to converge, while distributional match (KL divergence) saturates by N ~ 10 (87-95% of improvement across five model seeds); the specific thresholds are estimates for this setting, and it is