首页 > AI前沿 > TIDE: Reclaiming Lost Performance In On-Policy Reinforcement Learning

TIDE: Reclaiming Lost Performance In On-Policy Reinforcement Learning

arXiv机器学习 2026-11-13 17:03 13 阅读 查看原文
用户需要我作为专业的新闻排版工程师,将提供的纯文本新闻正文排版成语义化的HTML片段。 需要遵循的规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/``等文档级标签。
    
    提供的正文:
    Synchronous, colocated RL post-training of large language models (LLMs) today suffers from severe under-utilization due to long-tail rollouts. Inference engines fall idle waiting for long-tail rollouts or decode at low, inefficient batch sizes. To address this, we present TIDE, an RL training system that preserves on-policy training semantics while recovering resource utilization. TIDE does so by two complementary techniques: (1) dynamically repurposing rollout GPUs for training while long-tail rollouts complete and (2) dynamically migrating rollouts between inference engines to boost decoding throughput. TIDE automatically determines the right time to repurpose rollout resources through an adaptive migration policy, obtaining performance improvements that generalize across multiple applications and hardware configurations without requiring bespoke configuration. In our evaluation, TIDE provides end-to-end speedups of up to 1.36x over colocated systems, outperforming all evaluated on-policy baselines and achieving up to 97% of optimal throughput.
    
    分析正文内容,可以将其划分为几个部分:
    第一部分(问题描述):Synchronous, colocated RL post-training... inefficient batch sizes.
    第二部分(提出解决方案):To address this, we present TIDE... resource utilization.
    第三部分(具体技术):TIDE does so by two complementary techniques: (1) ... and (2) ...
    第四部分(自适应策略与评估结果):TIDE automatically determines... optimal throughput.
    
    排版设计:
    可以使用 `

    ` 或 `

    ` 来划分小节,虽然原文没有明确的小标题,但为了符合“长文用

    /

    划分小节”的要求,我可以添加几个小标题,但规则4说“不得增删改内容”。添加小标题算增加内容吗? 如果添加小标题,可能会被视为“增删改内容”。为了严格遵守“保留原文所有信息,不得增删改内容”,最好不添加原文没有的小标题文字。 那“长文用

    /

    划分小节”这个规则怎么满足?原文只有一段话。我可以把这段长文本按逻辑拆分成几个段落,或者不使用h2/h3,因为原文本身不是长文或者没有明显的章节。 或者,我可以不加h2/h3,直接使用p。因为原文只有一段,没有小节。 等等,如果原文只有一段,我也可以将其拆分为几个`

    `,表示不同的自然段落(虽然原文是一整段,但按句群拆分也可以,或者就保持一个`

    `)。 为了体现排版,我可以把技术部分(1)和(2)变成列表吗?原文是“(1) dynamically repurposing... and (2) dynamically migrating...”,我可以将其改为`

    • `,但这算不算修改内容?把内联的(1)...(2)改成列表,通常算作排版优化,语义没变。不过为了安全,保持原文的(1)和(2)在文本中,或者改成列表。 规则说“列表内容用
      • ”,原文中“(1)... and (2)...”其实是列表内容的平铺。改成`
        • `是合理的排版。 让我们仔细处理: 原文: Synchronous, colocated RL post-training of large language models (LLMs) today suffers from severe under-utilization due to long-tail rollouts. Inference engines fall idle waiting for long-tail rollouts or decode at low, inefficient batch sizes. To address this, we present TIDE, an RL training system that preserves on-policy training semantics while recovering resource utilization. TIDE does so by two complementary techniques: (1) dynamically repurposing rollout GPUs for training while long-tail rollouts complete and (2) dynamically migrating rollouts between inference engines to boost decoding throughput. TIDE automatically determines the right time to repurpose rollout resources through an adaptive migration policy, obtaining performance improvements that generalize across multiple applications and hardware configurations without requiring bespoke configuration. In our evaluation, TIDE provides end-to-end speedups of up to 1.36x over colocated systems, outperforming all evaluated on-policy baselines and achieving up to 97% of optimal throughput. 排版:

          Synchronous, colocated RL post-training of large language models (LLMs) today suffers from severe under-utilization due to long-tail rollouts. Inference engines fall idle waiting for long-tail rollouts or decode at low, inefficient batch sizes.

          To address this, we present TIDE, an RL training system that preserves on-policy training semantics while recovering resource utilization. TIDE does so by two complementary techniques:

          • dynamically repurposing rollout GPUs for training while long-tail rollouts complete
          • dynamically migrating rollouts between inference engines to boost decoding throughput

          TIDE automatically determines the right time to repurpose rollout resources through an adaptive migration policy, obtaining performance improvements that generalize across multiple applications and hardware configurations without requiring bespoke configuration. In our evaluation, TIDE provides end-to-end speedups of up to 1.36x over colocated systems, outperforming all evaluated on-policy baselines and achieving up to 97% of optimal throughput.

          检查是否增删改内容: 原文:“(1) dynamically repurposing rollout GPUs for training while long-tail rollouts complete and (2) dynamically migrating rollouts between inference engines to boost decoding throughput.” 改成列表后,去掉了“(1)”、“(2)”和“and”。这算不算删改内容? 如果严格来说,把“(1) A and (2) B”变成列表,去掉了数字和and,可能被判定为修改内容。 为了绝对安全,我不加列表,或者保留数字在li里。 比如:
          • (1) dynamically repurposing rollout GPUs for training while long-tail rollouts complete
          • (2) dynamically migrating rollouts between inference engines to boost decoding throughput
          把“and”去掉算不算修改?原文是“complete and (2) dynamically...”,在列表中,最后一项不需要“and”。这属于合理的HTML排版转换。 或者,我直接不拆成列表,保持原文的文本结构,只用`

          `和``。 规则3说“列表内容用

          • ”,如果原文没有明显的列表格式(虽然(1)...(2)...有列表意味),不强制使用。 但为了展现排版能力,把(1)(2)转为列表是很好的。 我们可以写:

            Synchronous, colocated RL post-training of large language models (LLMs) today suffers from severe under-utilization due to long-tail rollouts. Inference engines fall idle waiting for long-tail rollouts or decode at low, inefficient batch sizes.

            To address this, we present TIDE, an RL training system that preserves on-policy training semantics while recovering resource utilization. TIDE does so by two complementary techniques:

            • (1) dynamically repurposing rollout GPUs for training while long-tail rollouts complete
            • (2) dynamically migrating rollouts between inference engines to boost decoding throughput

            TIDE automatically determines the right time to repurpose rollout resources through an adaptive migration policy, obtaining performance improvements that generalize across multiple applications and hardware configurations without requiring bespoke configuration. In our evaluation, TIDE provides end-to-end speedups of up to 1.36x over colocated systems, outperform