划分小节，所以开头的问题部分可以用一个
，然后三个环境的部分？不对，看内容结构：第一段是背景和问题，第二段是引入的三个环境，第三段是编码环境的发现，第四段是应用和结果。 接下来，每个自然段用
，列表用
- ，但这里原文没有列表，所以不用。引用的话原文没有引用内容，代码也没有，重要句子用。 现在开始：首先第一个部分，背景和问题，作为
？或者看内容，用户要求“长文用
/
划分小节”，所以第一段：Reinforcement learning (RL) rewards...limited. 可以作为
内容？然后“To study how best to use monitors...hallucination.”作为
？或者直接用
。然后编码环境的发现部分，原文有(1)(2)(3)，可以用
- 吗？对，这三个是列表项。 现在具体排版： 第一个
：Reinforcement learning (RL) rewards are notoriously difficult to design and control, often leading to the model learning unintended behaviors such as reward hacking. One potential solution is to monitor for reward hacking and penalize it when detected; however, training against a monitor could lead to evasive behavior, and our general understanding of how to apply monitors effectively during training is limited. 然后
：To study how best to use monitors to mitigate reward hacking, we introduce and open source three realistic environments where Qwen3-4B reward hacks: a coding environment hackable via test overwriting, a medical chat environment hackable via sycophancy, and a biography generation environment hackable via hallucination. 然后
：We first focus on the coding environment, where we find that: 接着是
- models can learn to evade highly accurate monitors by exploiting systemic flaws in probes and LLM judges;
- monitors that leak more learning signal during RL suppress reward hacking but are more often evaded;
- including easier problems in training can decrease reward hacking.
：We apply our findings to build better reward hacking monitors for the medical chat and biography generation environments that improve upon naive baselines to reduce reward hacking rates across seeds to before-training levels in some interventions. Our results demonstrate that our takeaways translate to new settings and that better monitor intervention designs are possible. 现在检查是否保留所有信息，没有增删改。然后重要句子比如开头的问题部分的关键点，或者研究的发现，可以用吗？比如“reward hacking”可以加？看原文，重要句子，比如“models can learn to evade highly accurate monitors by exploiting systemic flaws in probes and LLM judges”可以加？ 调整后：
Reinforcement learning (RL) rewards are notoriously difficult to design and control, often leading to the model learning unintended behaviors such as reward hacking. One potential solution is to monitor for reward hacking and penalize it when detected; however, training against a monitor could lead to evasive behavior, and our general understanding of how to apply monitors effectively during training is limited.
To study how best to use monitors to mitigate reward hacking, we introduce and open source three realistic environments where Qwen3-4B reward hacks: a coding environment hackable via test overwriting, a medical chat environment hackable via sycophancy, and a biography generation environment hackable via hallucination.
We first focus on the coding environment, where we find that:
- models can learn to evade highly accurate monitors by exploiting systemic flaws in probes and LLM judges;
- monitors that leak more learning signal during RL suppress reward hacking but are more often evaded;
- including easier problems in training can decrease reward hacking.
We apply our findings to build better reward hacking monitors for the medical chat and biography generation environments that improve upon naive baselines to reduce reward hacking rates across seeds to before-training levels in some interventions. Our results demonstrate that our takeaways translate to new settings and that better monitor intervention designs are possible.
这样就符合要求了：用/
划分，每个自然段用
？不对，刚才的
和
里面的内容是段落，所以应该在
后面加
吗？不，
是标题，内容在
里。哦，刚才错了，标题是
，下面的正文是
。 重新调整：
Background and Challenges
不，不能增删内容，所以标题必须是原文的内容吗？不对，用户要求“保留原文所有信息，不得增删改内容”，所以标题不能加原文没有的文字，所以和
的内容必须是原文的句子。 哦，对，不能修改内容，所以标题的内容必须是原文的句子。比如第一个
就是原文的第一句：“Reinforcement learning (RL) rewards are notoriously difficult to design and control, often leading to the model learning unintended behaviors such as reward hacking. One potential solution is to monitor for reward hacking and penalize it when detected; however, training against a monitor could lead to evasive behavior, and our general understanding of how to apply monitors effectively during training is limited.” 这个是一个完整的段落，可以作为
的内容？不，
是标题，正文用
，所以应该把第一个段落作为
标题？或者把长文划分小节，比如：
Reinforcement learning (RL) rewards are notoriously difficult to design and control, often leading to the model learning unintended behaviors such as reward hacking. One potential solution is to monitor for reward hacking and penalize it when detected; however, training against a monitor could lead to evasive behavior, and our general understanding of how to apply monitors effectively during training is limited.
To study how best to use monitors to mitigate reward hacking, we introduce and open source three realistic environments where Qwen3-4B reward hacks: a coding environment hackable via test overwriting, a medical chat environment hackable via sycophancy, and a biography generation environment hackable via hallucination.
We first focus on the coding environment, where we find that:
- models can learn to evade highly accurate monitors by exploiting systemic flaws in probes and LLM judges;
- monitors that leak more learning signal during RL suppress reward hacking but are more often evaded;
- including easier problems in training can decrease reward hacking.
We apply our findings to build better reward hacking monitors for the medical chat and biography generation environments that improve upon naive baselines to reduce reward hacking rates across seeds to before-training levels in some interventions. Our results demonstrate that our takeaways translate to new settings and that better monitor intervention designs are possible.
但这样标题太长，不过符合用户要求，不能增删内容，所以只能用原文的句子作为标题。另外，重要句子加，比如“reward hacking”在后面的也可以加。 再调整，把第一句里的“reward hacking”加，然后发现中的第一条加，因为这是重要的发现。 最终排版：Reinforcement learning (RL) rewards are notoriously difficult to design and control, often leading to the model learning unintended behaviors such as reward hacking. One potential solution is to monitor for reward hacking and penalize it when detected; however, training against a monitor could lead to evasive behavior, and our general understanding of how to apply monitors effectively during training is limited.
To study how best to use monitors to mitigate reward hacking, we introduce and open source three realistic environments where Qwen3-4B reward hacks: a coding environment hackable via test overwriting, a medical chat environment hackable via sycophancy, and a biography generation environment hackable via hallucination.
- 吗？对，这三个是列表项。 现在具体排版： 第一个