首页 > AI前沿 > Minimal Superior Penalties: Characterising and Calibrating Decision-Aware Proper Scoring Rules

Minimal Superior Penalties: Characterising and Calibrating Decision-Aware Proper Scoring Rules

arXiv机器学习 2026-11-02 00:23 7 阅读 查看原文
首先,用户要求我把纯文本新闻正文排版成语义化的HTML片段。规则是: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    
    4. 保留原文所有信息,不得增删改内容。
    
    5. 不要输出``/``/``等文档级标签。
    
    正文是关于训练分类器的验证指标的。我需要分析正文,确定如何分段和使用语义化标签。
    
    正文内容:
    
    "Training a classifier means letting a validation metric decide which model is kept and when training stops. A natural requirement on that metric is superiority: every correctly classified forecast should score better than every incorrectly classified one, however close the underlying probabilities are. Neither the Brier score nor the logarithmic score satisfies this on its own. We show that superiority is governed by the separation between two endpoints -- the supremum of the score on correct predictions and the infimum on incorrect predictions -- and derive both in closed form for each score. Their difference is the infimal penalty, the greatest lower bound on the penalty needed to restore superiority. Prior work restores superiority with a penalty calibrated from the first endpoint alone; for the Brier base, our infimal penalty is smaller than that published penalty by a factor ranging from 4.00 at three classes to 2.11 at twenty. We further embed the published and infimal penalties in a one-parameter family and show that, under a single-valued tie convention, every member of the family is strictly proper. In a pre-specified two-dataset case study, mean selected-model macro-F1 rises with the penalty in the reported ten-class regimes, though not distinguishably from noise once seed/selection-mode nesting is accounted for; a fifth regime at forty-seven classes does not reproduce even this descriptive pattern. The intended no-superiority baseline is algebraically a rescaling of the family under test, so the experiment cannot attribute the observed pattern specifically to superiority. When superiority itself is the objective, any penalty strictly above the infimal value suffices; choosing a larger penalty is a separate calibration trade-off rather than a requirement for the guarantee."
    
    现在,我需要将这个文本分成逻辑段落。看内容:
    
    - 第一段:介绍训练分类器和验证指标的要求。
    
    - 然后提到Brier score和logarithmic score不满足要求。
    
    - 接着展示superiority如何由两个端点控制,并推导封闭形式。
    
    - 定义infimal penalty。
    
    - 比较之前的penalty和现在的。
    
    - 嵌入到一个参数族中。
    
    - 案例研究部分。
    
    - 最后关于选择penalty的建议。
    
    所以,我可以这样分段:
    
    1. 开头介绍:从"Training a classifier..." 到 "Neither the Brier score nor the logarithmic score satisfies this on its own."
    
    2. 然后展示结果:从"We show that superiority..." 到 "derive both in closed form for each score."
    
    3. 定义infimal penalty:从"Their difference is the infimal penalty..." 到 "needed to restore superiority."
    
    4. 比较prior work:从"Prior work restores superiority..." 到 "at three classes to 2.11 at twenty."
    
    5. 参数族部分:从"We further embed the published and infimal penalties..." 到 "every member of the family is strictly proper."
    
    6. 案例研究:从"In a pre-specified two-dataset case study..." 到 "does not reproduce even this descriptive pattern."
    
    7. 解释实验:从"The intended no-superiority baseline..." 到 "specifically to superiority."
    
    8. 最后建议:从"When superiority itself is the objective..." 到结束。
    
    用户说"长文用

    /

    划分小节",所以如果文本较长,我需要添加标题。但这里文本不是太长,可能不需要所有都用标题。规则是"长文用",所以或许只对主要部分添加标题。 用户说"每个自然段落用

    包裹",所以每个逻辑段落应该用

    标签。 另外,有重要句子可用。我可以找出一些关键句子加粗。 例如,重要句子如:"We show that superiority is governed by the separation between two endpoints..." 或者 "Their difference is the infimal penalty..." 等。 但用户说"重要句子可用",意思是可以选择性地使用,不是必须。所以我需要判断哪些句子重要。 保留所有信息,不得增删改内容,所以我不能改变文本内容,只能添加HTML标签。 现在,列出所有段落: - 段落1: "Training a classifier means letting a validation metric decide which model is kept and when training stops. A natural requirement on that metric is superiority: every correctly classified forecast should score better than every incorrectly classified one, however close the underlying probabilities are. Neither the Brier score nor the logarithmic score satisfies this on its own." - 段落2: "We show that superiority is governed by the separation between two endpoints -- the supremum of the score on correct predictions and the infimum on incorrect predictions -- and derive both in closed form for each score." - 段落3: "Their difference is the infimal penalty, the greatest lower bound on the penalty needed to restore superiority." - 段落4: "Prior work restores superiority with a penalty calibrated from the first endpoint alone; for the Brier base, our infimal penalty is smaller than that published penalty by a factor ranging from 4.00 at three classes to 2.11 at twenty." - 段落5: "We further embed the published and infimal penalties in a one-parameter family and show that, under a single-valued tie convention, every member of the family is strictly proper." - 段落6: "In a pre-specified two-dataset case study, mean selected-model macro-F1 rises with the penalty in the reported ten-class regimes, though not distinguishably from noise once seed/selection-mode nesting is accounted for; a fifth regime at forty-seven classes does not reproduce even this descriptive pattern." - 段落7: "The intended no-superiority baseline is algebraically a rescaling of the family under test, so the experiment cannot attribute the observed pattern specifically to superiority." - 段落8: "When superiority itself is the objective, any penalty strictly above the infimal value suffices; choosing a larger penalty is a separate calibration trade-off rather than a requirement for the guarantee." 这些段落都是自然段落。 现在,考虑是否添加标题。用户说"长文用

    /

    划分小节",但这里文本不长,可能不需要。但为了语义化,我可以添加一些小标题。 例如,可以分成几个小节: - 介绍部分 - 方法论 - 结果比较 - 案例研究 - 结论 但用户说"划分小节",所以我可以添加

    或

    来组织内容。 规则是"长文用",所以或许只当文本很长时。但这里文本适中,我可以添加一些标题来增强语义。 为了安全,我可以添加标题。 看内容,有明确的部分: - 从"We show that..." 开始是展示结果 - 然后"Prior work..." 是相关工作 - "We further embed..." 是进一步工作 - "In a pre-specified..." 是案例研究 - 最后是结论 所以,我可以添加

    标题如: -

    介绍

    -

    方法论

    -

    结果比较

    -

    案例研究

    -

    结论

    但用户说"语义化的HTML片段",并且"保留原文所有信息,不得增删改内容",所以我不能添加新内容。标题是添加的,但用户允许用

    /

    划分小节,所以可以添加标题来组织,但不能改变内容。 用户说"不得增删改内容",但添加标题是结构上的,不是内容上的。规则是"划分小节",所以应该是允许的。