首页 > AI前沿 > ModHarness-RL: Recasting Harnesses as Learnable and Evolving Programs for Agentic RL

ModHarness-RL: Recasting Harnesses as Learnable and Evolving Programs for Agentic RL

arXiv自然语言 2026-11-14 02:45 12 阅读 查看原文
用户需要我扮演专业的新闻排版工程师,将给定的纯文本新闻正文排版成语义化的HTML片段。 需要遵循以下规则: 1. 只输出HTML片段本身,不要Markdown围栏,不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/``等文档级标签。
    
    分析正文内容:
    "Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives. Most existing approaches train under a hand-written harness that stays fixed throughout. However, an effective harness depends on the domain, the data and the model, and keeps changing as the model trains. We introduce ModHarness-RL, which recasts the harness as programs that are learned from data and evolve as the model trains with RL. ModHarness-RL records the interaction as structured State, a set of typed blocks, and two core components of the harness are programs over these blocks, a Compression program that assembles the accumulated history into the model's context and a Steering program that supplies guidance when behavior deviates. During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model's behavior shifts. We study CodeSearch in depth and further analyze ALFWorld and BIRD SQL, with models from 1.5B to 8B. The proposed Compression and Steering mechanisms improve held-out performance over the fixed-harness baseline in most settings. These results suggest that a harness for agentic RL can be learned from the task's data and evolved with the model."
    
    这段文字是一个学术新闻/摘要,没有明显的列表、引用或代码。它是一段连续的文本。
    为了符合规则2“长文用

    /

    划分小节”,虽然原文没有明确的小标题,但我可以通过提取关键概念来划分小节,或者如果认为这只是一段,就只用`

    `。但规则说“长文用

    /

    划分小节”,这段文字算长文,可以合理划分小节,比如:

    背景与挑战

    Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives. Most existing approaches train under a hand-written harness that stays fixed throughout. However, an effective harness depends on the domain, the data and the model, and keeps changing as the model trains.

    ModHarness-RL 的核心方法

    We introduce ModHarness-RL, which recasts the harness as programs that are learned from data and evolve as the model trains with RL. ModHarness-RL records the interaction as structured State, a set of typed blocks, and two core components of the harness are programs over these blocks, a Compression program that assembles the accumulated history into the model's context and a Steering program that supplies guidance when behavior deviates.

    训练过程与实验验证

    During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model's behavior shifts. We study CodeSearch in depth and further analyze ALFWorld and BIRD SQL, with models from 1.5B to 8B.

    实验结果与结论

    The proposed Compression and Steering mechanisms improve held-out performance over the fixed-harness baseline in most settings. These results suggest that a harness for agentic RL can be learned from the task's data and evolved with the model.

    但是规则4要求“保留原文所有信息,不得增删改内容”。添加小标题算不算增删改内容?严格来说,添加小标题是增加了原文没有的文本(如“背景与挑战”)。为了绝对安全,不增加任何原文没有的文字,我可以不添加额外的小标题文本,或者把原有的句子作为小节标题?但原文没有句子适合做标题。 或者,我直接将其作为一个长段落,用`

    `包裹,里面对重要句子使用``。因为原文本身就是一段,没有分节标识。如果强行加小标题,就会引入原文没有的词汇,违反规则4“不得增删改内容”。 不过规则2说“长文用

    /

    划分小节”,如果不划分,可能会被认为没有遵循规则。有没有办法在不增加新词的情况下划分? 比如,把第一句话作为`

    `?

    Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives.

    这改变了原文的排版,但保留了文字。不过这会改变原文的语义结构。 其实,原文就是一段摘要。直接用一个`

    `包裹,并在关键处使用``是最稳妥的,完全遵守“不得增删改内容”。但为了体现“长文用

    /

    划分小节”的规则,我可以将原文按逻辑切分成几个部分,每部分用`

    `,并加上`

    `,但`

    `的文本必须来自原文,或者干脆不加标题,只用`

    `。 再仔细看规则:“每个自然段落用

    包裹,长文用

    /

    划分小节”。原文是一整段。如果把它拆成几个自然段,其实也是改变了原文的段落结构(原文是一个自然段)。 “保留原文所有信息,不得增删改内容”通常指文字内容不增删改,排版上的合理切分是允许的。为了稳妥,我直接将整段文字放在一个`

    `中,对重要名词或结论使用``。如果必须用小标题,我可以把原文中的关键短语提取出来做小标题,但这有风险。 或者,我不使用小标题,因为这段文字在原文中可能只是一个段落(abstract),不算“长文”。直接输出一个`

    `。 为了展示对规则的运用,我可以这样:

    Agentic reinforcement learning (RL) increasingly relies on a harness, the scaffold around the model that decides what it sees of its own interaction and what guidance it receives. Most existing approaches train under a hand-written harness that stays fixed throughout. However, an effective harness depends on the domain, the data and the model, and keeps changing as the model trains.

    We introduce ModHarness-RL, which recasts the harness as programs that are learned from data and evolve as the model trains with RL. ModHarness-RL records the interaction as structured State, a set of typed blocks, and two core components of the harness are programs over these blocks, a Compression program that assembles the accumulated history into the model's context and a Steering program that supplies guidance when behavior deviates.

    During RL training, candidate programs are proposed from recent trajectories, verified by execution, and replayed over past trajectories, so the harness tracks how the model's behavior shifts. We study CodeSearch in depth and further analyze ALFWorld and BIRD SQL, with models from 1.5B to 8B. The proposed Compression and Steering mechanisms improve held-out performance over the fixed-harness baseline in most settings. These results suggest that a harness for agentic RL can be learned from the task's data and evolved with the model.

    这样切分了三个自然段落,没有添加任何原文没有的标题文字,完全遵守“不得增删改内容”。同时使用了`

    `和``。没有列表、引用、代码,所以不需要使用`

      `等。 检查是否满足所有规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释; -> 满足 2. 每个自然段落用

      包裹,长文用

      /

      划分小节; -> 这里没有用h2/h3,因为添加标题会违反规则4。如果非要用,可以用原文的句子,但没必要。或者我可以把三个部分分别加上h3,标题