首页 > AI前沿 > LACCO: Local Accumulation with Computation--Communication Overlap

LACCO: Local Accumulation with Computation--Communication Overlap

arXiv机器学习 2026-11-13 18:21 4 阅读 查看原文
用户要求我作为专业的新闻排版工程师,将一段纯文本的英文新闻正文排版成语义化的HTML片段。 需要遵循的规则: 1. 只输出HTML片段本身,不要Markdown围栏(```html ... ```),不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/``等文档级标签。
    
    分析正文内容:
    这段英文是关于训练语言模型的,主要介绍了一种名为LACCO的新算法。
    正文可以分为几个部分:
    - 第一句到"drift.":介绍背景和现有方法的问题。
    - "We propose LACCO...":介绍提出的方法。
    - "We establish convergence...":理论保证。
    - "Experiments on language-model...":实验结果和结论。
    
    考虑到规则2要求“每个自然段落用

    包裹,长文用

    /

    划分小节”,这段文本可以划分为几个小节,或者直接作为几个段落。由于它只是一段较长的学术新闻/摘要,我可以使用`

    `来划分小节,使其结构更清晰,或者直接用`

    `。为了体现“长文用

    /

    划分小节”,我可以加一些小标题,但规则4要求“不得增删改内容”,加小标题算“增”吗? 规则4:“保留原文所有信息,不得增删改内容”。添加小标题可能会被视为增加了内容。因此,最安全的做法是不添加原文没有的小标题,而是直接将每个自然段用`

    `包裹。或者,原文本身是一整段,我把它分成几个`

    `,按句子或逻辑块分段,但不添加标题。 等等,原文是一整段文本。如果我把它拆成多个`

    `,算不算增删改?通常排版时将长段落拆分是可以的,只要文字内容不改变。但为了严格遵守“不得增删改内容”,我可以直接用一个`

    `包裹整段,或者按句子逻辑拆成几个`

    `,不添加任何标题。 不过,规则提到“长文用

    /

    划分小节”,如果这段文本本身没有小节标题,我强行加就违反了“不得增内容”。所以这里不使用`

    /

    `,直接使用`

    `。 让我们把原文按逻辑拆分为几个段落,或者就保持为一个段落。为了排版美观,可以按句子组合拆分为几个`

    `。 原文: "Training language models across low-bandwidth networks requires reducing synchronization overhead. Local optimization methods such as DiLoCo communicate only after several local steps, but outer gradient averaging and the outer update remain on the critical path. Existing computation-communication overlap methods avoid this stall through delayed or replica-local updates, which can degrade convergence or cause replicas to drift." "We propose LACCO (Local Accumulation with Computation-Communication Overlap), a two-stage algorithm that overlaps global communication with local computation while applying globally aggregated updates shared by every replica, without introducing staleness in the outer update." "We establish convergence guarantees for smooth non-convex objectives." "Experiments on language-model pre-training with FineWeb show that LACCO consistently outperforms communication-overlapped baselines such as Eager and Delayed DiLoCo. These gains remain robust as model size scales up to 0.5B parameters and across a range of communication intervals and numbers of replicas up to 16 workers, while incurring only negligible-or fully overlappable-computational or memory overhead, which demonstrates the strength of LACCO." 可以将其分为4个`

    `标签。对于“重要句子可用``”,可以将核心方法或结论加粗。例如“We propose LACCO...”或者“demonstrates the strength of LACCO”等。这里我选择将“LACCO”相关的核心句子部分用``标记,比如第一段的“Existing computation-communication overlap methods...”或者第二段的“We propose LACCO...”。适度加粗即可。 排版草稿:

    Training language models across low-bandwidth networks requires reducing synchronization overhead. Local optimization methods such as DiLoCo communicate only after several local steps, but outer gradient averaging and the outer update remain on the critical path. Existing computation-communication overlap methods avoid this stall through delayed or replica-local updates, which can degrade convergence or cause replicas to drift.

    We propose LACCO (Local Accumulation with Computation-Communication Overlap), a two-stage algorithm that overlaps global communication with local computation while applying globally aggregated updates shared by every replica, without introducing staleness in the outer update.

    We establish convergence guarantees for smooth non-convex objectives.

    Experiments on language-model pre-training with FineWeb show that LACCO consistently outperforms communication-overlapped baselines such as Eager and Delayed DiLoCo. These gains remain robust as model size scales up to 0.5B parameters and across a range of communication intervals and numbers of replicas up to 16 workers, while incurring only negligible-or fully overlappable-computational or memory overhead, which demonstrates the strength of LACCO.

    检查规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 -> 是。 2. 每个自然段落用

    包裹,长文用

    /

    划分小节。 -> 这里没有小节标题,所以只用

    ,合理。 3. 列表内容用

    • ,引用用
      ,代码用
      ,重要句子可用。 -> 使用了。没有列表、引用、代码,所以不需要那些标签。
      4. 保留原文所有信息,不得增删改内容。 -> 文字完全一致,只是加了HTML标签和换行分段。分段是否算“改内容”?通常排版允许合理分段。如果不分段,直接一个

      包含所有文本,也是完全符合“不增删改”的。为了绝对安全,我可以直接用一个

      包裹整段,或者按原样的一整段输出。但规则说“每个自然段落用

      包裹”,原文是一整段,所以用一个

      包裹整段最符合“不增删改内容”且“每个自然段落用

      包裹”。 不过,把长段落在排版时拆分是常见的排版动作。为了稳妥,我把原文视为一个自然段,用一个`

      `包裹,内部用``加粗重要句子。或者拆成几个`

      `,因为排版工程师经常会做段落重组。考虑到“不得增删改内容”,不改变文本顺序和文字,只是增加HTML标签,拆成多个`

      `是合理的排版。我还是拆成几个逻辑段落吧,或者就保持一个段落。 其实保持一个`

      `最安全,避免被判定为“私自拆分段落属于修改内容”。 “Training language models ... strength of LACCO.” 这是一个自然段落。 所以:

      Training language models across low-bandwidth networks requires reducing synchronization overhead. Local optimization methods such as DiLoCo communicate only after several local steps, but outer gradient averaging and the outer update remain on the critical path. Existing computation-communication overlap methods avoid this stall through delayed or replica-local updates, which can degrade convergence or cause replicas to drift. We propose LACCO (Local Accumulation with Computation-Communication Overlap), a two-stage algorithm that overlaps global communication with local computation while applying globally aggregated updates shared by every replica, without introducing staleness in the outer update. We establish convergence guarantees for smooth non-convex objectives. Experiments on language-model pre-training with FineWeb show that LACCO consistently outperforms communication-overlapped baselines such as Eager and Delayed DiLoCo. These gains remain robust as model size scales up to 0.5B parameters and across a range of communication intervals and numbers of replicas up to 16 workers, while incurring only negligible-or fully overlappable-computational or memory overhead, which demonstrates the strength of LACCO.

      这样完全保留了原文的段落结构(原文就是一整段