首页 > AI前沿 > Blind Spots Travel, Knowledge Stays Home: What RL Inherits from an LLM Judge

Blind Spots Travel, Knowledge Stays Home: What RL Inherits from an LLM Judge

arXiv自然语言 2026-11-20 13:00 32 阅读 查看原文
用户要求将提供的纯文本新闻正文排版成语义化的HTML片段。 需要遵循以下规则: 1. 只输出HTML片段本身,不要Markdown围栏(如```html),不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节(这里文本不算特别长,但可以合理划分小节或直接分段)。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/``等文档级标签。
    
    分析正文内容:
    正文是一段关于LLM judges作为奖励模型在RL训练中的表现的研究摘要/新闻。
    可以将其分为几个逻辑段落,或者加上小标题。
    
    原文:
    LLM judges serve as rewards where no answer key exists, in the hope that their knowledge passes to the trained model. In 814 RL runs that teach 7-8B models to answer closed-book questions or abstain, we ask what passes. What a grader knows about single answers stays home. A wrong answer that a judge rewards is memorised for that one question. A model trained with the answer key, or with a far stronger judge, chooses which new questions to answer no better than the untrained model does by sampling 32 answers and abstaining where they disagree; the grader's knowledge adds about one corrected answer per hundred questions. What a grader believes about a kind of question travels. A judge that rejects eleven of the twelve correct training answers about capitals of former states teaches the model to refuse 98% of such capitals on new questions, a loss that accuracy and abstention rates do not reveal. Changing only those twelve verdicts by hand moves the refusal from 5% to 90%, even for a grader that accepts as many answers as the answer key; at that leniency, two of four judges already cost 38-42% of these capitals. The capitals are not forgotten: with "I don't know" blocked, the model answers 94% of them correctly. The grader therefore needs no knowledge of its own: rewarding the model for giving the answer that most of its own untrained samples agree on, with no labels and no judge, earns the answer key's reward and keeps the category.
    
    划分段落和结构:
    可以加一个小标题,比如 `

    LLM Judges in RL Training

    `,然后分几个段落。 段落1:LLM judges serve as rewards... one corrected answer per hundred questions. 段落2:What a grader believes about a kind of question travels. A judge that rejects... do not reveal. 段落3:Changing only those twelve verdicts... of these capitals. 段落4:The capitals are not forgotten... keeps the category. 或者直接按自然段分: 原文看起来是一整个自然段,但为了排版好看,可以根据句群拆分成多个`

    `。 1. LLM judges serve as rewards where no answer key exists, in the hope that their knowledge passes to the trained model. In 814 RL runs that teach 7-8B models to answer closed-book questions or abstain, we ask what passes. 2. What a grader knows about single answers stays home. A wrong answer that a judge rewards is memorised for that one question. A model trained with the answer key, or with a far stronger judge, chooses which new questions to answer no better than the untrained model does by sampling 32 answers and abstaining where they disagree; the grader's knowledge adds about one corrected answer per hundred questions. 3. What a grader believes about a kind of question travels. A judge that rejects eleven of the twelve correct training answers about capitals of former states teaches the model to refuse 98% of such capitals on new questions, a loss that accuracy and abstention rates do not reveal. 4. Changing only those twelve verdicts by hand moves the refusal from 5% to 90%, even for a grader that accepts as many answers as the answer key; at that leniency, two of four judges already cost 38-42% of these capitals. 5. The capitals are not forgotten: with "I don't know" blocked, the model answers 94% of them correctly. 6. The grader therefore needs no knowledge of its own: rewarding the model for giving the answer that most of its own untrained samples agree on, with no labels and no judge, earns the answer key's reward and keeps the category. 加上重要的加粗(重要句子可用``): 例如: What a grader knows about single answers stays home. What a grader believes about a kind of question travels. 检查是否增删改内容:必须严格保留原文。 构建HTML:

    LLM Judges as Rewards in RL Training

    LLM judges serve as rewards where no answer key exists, in the hope that their knowledge passes to the trained model. In 814 RL runs that teach 7-8B models to answer closed-book questions or abstain, we ask what passes.

    What a grader knows about single answers stays home. A wrong answer that a judge rewards is memorised for that one question. A model trained with the answer key, or with a far stronger judge, chooses which new questions to answer no better than the untrained model does by sampling 32 answers and abstaining where they disagree; the grader's knowledge adds about one corrected answer per hundred questions.

    What a grader believes about a kind of question travels. A judge that rejects eleven of the twelve correct training answers about capitals of former states teaches the model to refuse 98% of such capitals on new questions, a loss that accuracy and abstention rates do not reveal.

    Changing only those twelve verdicts by hand moves the refusal from 5% to 90%, even for a grader that accepts as many answers as the answer key; at that leniency, two of four judges already cost 38-42% of these capitals.

    The capitals are not forgotten: with "I don't know" blocked, the model answers 94% of them correctly.

    The grader therefore needs no knowledge of its own: rewarding the model for giving the answer that most of its own untrained samples agree on, with no labels and no judge, earns the answer key's reward and keeps the category.

    检查规则: 1. 只输出HTML片段本身,无Markdown,无解释。 -> 是 2. 每个自然段落用

    包裹,长文用

    /

    划分小节。 -> 是 3. 列表、引用、代码、加粗。 -> 加粗了重要句子。 4. 保留原文所有信息,不得增删改。 -> 检查每个词。 原文:LLM judges serve as rewards where no answer key exists, in the hope that their knowledge passes to the trained model. In 814 RL runs that teach 7-8B models to answer closed-book questions or abstain, we ask what passes. HTML: LLM judges serve as rewards where no answer key exists, in the hope that their knowledge passes to the trained model. In 814 RL runs that teach 7-8B models to answer closed-book questions or abstain, we ask what passes. (一致) 原文:What a grader knows about single answers stays home. A wrong answer that a judge rewards is memorised for that one question. A model trained with the answer key, or with a far stronger judge, chooses which new questions to answer no better than the untrained model does by sampling 32 answers and abstaining where they disagree; the grader's knowledge adds about one corrected answer per hundred questions. HTML: What a grader knows about single answers stays home. A wrong answer that a judge rewards is memorised for that one question. A model trained with the answer key, or with a far stronger judge, chooses which new questions to answer no better than the untrained model does by sampling 32 answers and abstaining where they disagree; the grader's knowledge adds about one corrected answer per hundred questions. (一致) 原文:What a grader believes about a kind of question travels. A judge that rejects eleven of the twelve correct training answers about capitals of former states teaches the model to refuse 98% of such capitals on new questions, a loss that accuracy and abstention rates do not reveal.