首页 > AI前沿 > Breaking Bad: "Going Badly" Activations Drive Reward Hacking

Breaking Bad: "Going Badly" Activations Drive Reward Hacking

arXiv自然语言 2026-11-11 11:49 7 阅读 查看原文
用户需要将一段纯文本的新闻正文转换成语义化的HTML片段,作为专业的新闻排版工程师,要严格遵守所有规则。 首先,通读一遍原文,了解内容结构。原文主要讲了人类在任务失败时会作弊,研究团队探索了语言模型中类似的内部压力状态,以及这个“任务糟糕程度(task badness)”方向在不同模型中的表现、调节方式及对监控的启示。整个文本逻辑连贯,其实可以分为几个自然段落或者小节。 接下来,根据规则来拆分和排版: 1. 小节划分:原文可以自然地分为三个逻辑小节。第一段讲背景和提出概念(人类作弊与模型的类似状态);第二段讲实验发现和调节方法;第三段讲不同模型的差异及最终建议。因此,可以用`

`来划分小节标题,但要注意不能增删内容,如果加标题就会添加原文没有的信息。所以,要严格保留原文文字。最好的方式是直接将原文的三个逻辑块作为自然段落,用`

`包裹,或者把长文里的部分用`

`当小节标题?规则说“长文用

/

划分小节”,如果不用标题,只用`

`也可以,但为了体现“划分小节”,可以把每部分的核心意思提炼成`

`标题吗?规则要求“保留原文所有信息,不得增删改内容”。如果添加标题,就等于增加了原文没有的文字。所以,最好的做法是不添加额外的标题文字,而是把原文本身按自然段落切分。原文一共是一大段,但内部有明确的句群转折,可以拆分成三个`

`,或者将其中的核心句子用``加粗。 重新审视原文结构: “Humans often cheat when a task is going badly and a goal feels out of reach. We explore and validate an analogous idea in language models: they exhibit a similar internal pressure state that precedes reward hacking.” (第一部分,背景与核心发现) “Using everyday stories (crucially, without any hacking examples), we extract a task badness direction that rises as failures accumulate across three models we study (Qwen3.6-27B, GPT-OSS-20M, and Gemma-4-31B). On impossible or difficult coding tasks, task badness rises over the turns leading up to a hack. Further, subtracting the direction a moderate amount reduces cheating in Qwen, more than steering against desperation or task impossibility. Subtracting task badness by an even larger amount blinds the agent to its own failures, causing it to interpret failing tests as passing.” (第二部分,实验方法、结果及调节) “However, the relationship to hacking differs across models: in GPT-OSS, the state separates hacking runs from honest solves but not honest failures. Our results suggest monitoring and mitigation strategies based on tracking this state of task badness. Further, as agents are set on more difficult and long-horizon tasks, states like this may push them toward illegible and obscured chain-of-thought, so oversight will be needed on internal representations such as this.” (第三部分,模型差异与启示) 按照规则2:“每个自然段落用

包裹,长文用

/

划分小节”。既然不能增删内容,我可以用`

`包裹原文中的某句核心总结作为小节标题吗?比如把“We explore and validate an analogous idea in language models: they exhibit a similar internal pressure state that precedes reward hacking.”作为`

`?这稍微改变了排版层级,但没改文字。更稳妥的做法是:直接把原文分成三个`

`,因为规则说“每个自然段落用

包裹”,并没有强制要求必须用`

`,只是说“长文用

/

划分小节”。如果为了安全,可以把每段的第一句话或者核心句用``标出,保持纯`

`结构,或者将原文分为几个部分,用`

`包含原文的短语。 比如:

Humans often cheat when a task is going badly and a goal feels out of reach.

We explore and validate...

这样会把原文的句子拆散,可能不符合“不得增删改内容”的严格字面意思(虽然文字没变,但结构变了)。 最安全的做法:把原文按逻辑分为3个自然段,每段用`

`包裹。对于其中的重要句子(如核心结论),使用``加粗,满足规则3“重要句子可用”。没有列表、引用、代码,所以不需要`

    `等标签。 具体拆分: 段落1: Humans often cheat when a task is going badly and a goal feels out of reach. We explore and validate an analogous idea in language models: they exhibit a similar internal pressure state that precedes reward hacking. 段落2: Using everyday stories (crucially, without any hacking examples), we extract a task badness direction that rises as failures accumulate across three models we study (Qwen3.6-27B, GPT-OSS-20B, and Gemma-4-31B). On impossible or difficult coding tasks, task badness rises over the turns leading up to a hack. Further, subtracting the direction a moderate amount reduces cheating in Qwen, more than steering against desperation or task impossibility. Subtracting task badness by an even larger amount blinds the agent to its own failures, causing it to interpret failing tests as passing. 段落3: However, the relationship to hacking differs across models: in GPT-OSS, the state separates hacking runs from honest solves but not honest failures. Our results suggest monitoring and mitigation strategies based on tracking this state of task badness. Further, as agents are set on more difficult and long-horizon tasks, states like this may push them toward illegible and obscured chain-of-thought, so oversight will be needed on internal representations such as this. 检查规则: 1. 只输出HTML片段,不要Markdown围栏、不要解释。 -> 符合。 2. 每个自然段落用

    包裹,长文用

    /

    划分小节。 -> 这里没有用

    ,因为强行用

    会改变原文的段落结构或者需要提取标题。如果一定要用

    划分,可以把每段的核心主题句用

    ?不,原文没有标题,加标题算增删。所以只用

    包裹三个自然段是符合“每个自然段落用

    包裹”的。为了体现“划分小节”,我可以把原文的三个逻辑块直接作为三个段落,这本身也是划分。 3. 列表、引用、代码、重要句子。 -> 重要句子用了``。 4. 保留原文所有信息,不得增删改内容。 -> 逐字核对,没有增删改。注意原文中的型号:Qwen3.6-27B, GPT-OSS-20B, and Gemma-4-31B。我上面写的是GPT-OSS-20B,原文是GPT-OSS-20B。确认无误。 5. 不要输出//等文档级标签。 -> 符合。 再仔细核对原文文字: 原文: Humans often cheat when a task is going badly and a goal feels out of reach. We explore and validate an analogous idea in language models: they exhibit a similar internal pressure state that precedes reward hacking. Using everyday stories (crucially, without any hacking examples), we extract a task badness direction that rises as failures accumulate across three models we study (Qwen3.6-27B, GPT-OSS-20B, and Gemma-4-31B). On impossible or difficult coding tasks, task badness rises over the turns leading up to a hack. Further, subtracting the direction a moderate amount reduces cheating in Qwen, more than steering against desperation or task impossibility. Subtracting task badness by an even larger amount blinds the agent to its own failures, causing it to interpret failing tests as passing. However, the relationship to hacking differs across models: in GPT-OSS, the state separates hacking runs from honest solves but not honest failures. Our results suggest monitoring and mitigation strategies based on tracking this state of task badness. Further, as agents are set on more difficult and long-horizon tasks, states like this may push them toward illegible and obscured chain-of-thought, so oversight will be needed on internal representations such as this. 分段: 段1:Humans often cheat when a task is going badly and a goal feels