首页 > AI前沿 > Architectures and KV Replay for Depth-Recurrent Language Models

Architectures and KV Replay for Depth-Recurrent Language Models

arXiv机器学习 2026-11-14 23:05 19 阅读 查看原文
用户现在需要我把给定的新闻正文排版成语义化HTML片段,遵循要求:每个自然段落用

,这里内容都是一段正文?不对,看内容是几个研究部分?不对,先看原文结构,首先开头是讲depth recurrence的问题,然后讲研究的几个方法,然后结果。首先,原文的内容,第一部分是讲depth recurrence的机制和存在的问题,然后接下来讲研究的几个方法:首先研究looped hybrid,然后是Parcae式架构,然后是looped Transformer with recurrent KV replay,然后是研究发现。不对,原文的段落其实是连贯的?不对,用户给的正文是一整段?不对,仔细看用户给的正文:是一整段?不,原文是: 第一句:Depth recurrence increases...during autoregressive decoding. 然后第二句:We study how to reduce...framework. 然后第三部分:We first study a looped hybrid...retained. 然后第四部分:We then restrict recurrence...wrapper for the looped hybrid. 然后第五部分:Finally, we train a looped Transformer...inference. 然后第六部分:We find that looped hybrids...for each iteration. 那需要合理划分小节吗?不对,规则说长文用h2/h3划分小节,这里的内容是研究的不同方法部分,所以可以划分h2小节。首先,开头的问题和方法总述用p?不对,先整理: 首先,开头介绍背景和研究目标的部分,用p?然后“研究方法”部分,分三个小节?不对,原文的We first, We then, Finally 是三个方法,然后是结果。那结构: 首先,第一部分(背景与研究目标)用p:

Depth recurrence increases effective model depth by repeatedly applying a fixed computation block. However, sharing attention parameters does not share the key-value (KV) states generated at different recurrence iterations, which causes memory growth with recurrence depth during autoregressive decoding. We study how to reduce this memory cost through architectural and cache-sharing approaches within a common depth-recurrent framework.

然后,研究方法部分用h2,然后分三个h3?不对,或者:

研究方法

循环混合架构

We first study a looped hybrid that repeats a stack of Transformer layers, with recurrent mLSTM or Gated DeltaNet blocks replacing most attention layers and periodic Transformer attention anchors retained.

Parcae式架构

We then restrict recurrence to a dedicated recurrent workspace in a Parcae-style architecture and test its stabilising adapter as a wrapper for the looped hybrid.

带循环KV重放的循环Transformer

Finally, we train a looped Transformer with recurrent KV replay, where only selected recurrence iterations write KV states that later iterations reuse. This trains the model to operate with the same shared KV topology used during inference.

然后研究结果用h2,然后p:

研究结论

We find that looped hybrids require sufficiently frequent attention anchors, with performance depending strongly on the recurrent mixer. For cross-iteration KV sharing, recurrent KV replay allows multiple recurrence iterations to reuse the same KV states while retaining performance close to the full looped Transformer with separate KV caches for each iteration.

不对,等一下,规则说“保留原文所有信息,不得增删改内容”,所以不能添加小节标题的内容?哦对!不能增删改,所以h2/h3的标题不能用自己加的,那怎么办?哦,原文里没有小节标题,那是不是不需要划分h2/h3?不对,看规则是“长文用

/

划分小节”,但如果原文没有小节标题,那怎么划分?不对,再看原文,原文的内容是: 1. 介绍depth recurrence的问题和研究目标 2. 第一个研究内容:looped hybrid 3. 第二个研究内容:Parcae式架构 4. 第三个研究内容:looped Transformer with recurrent KV replay 5. 研究发现 那如果要划分小节,是不是不能加标题,那不对,规则说“不得增删改内容”,所以不能自己加标题。那是不是把这些内容都用p?不对,或者看原文的结构,其实都是自然段落?不对,用户给的原文是一整段,但可以拆分成几个p?不对,规则说“每个自然段落用

包裹”,那原文的每个自然段落,用户给的原文是一个自然段落吗?不,仔细看用户给的正文,是连续的一段,但里面的几个研究部分是逻辑上的小节,但不能加标题,那怎么办?哦,不对,规则说“长文用

/

划分小节”,但如果原文没有小节标题,那是不是不需要用?或者是不是把原文的内容按照逻辑拆分,每个逻辑部分用p,重要句子用strong?不对,再看规则,不能增删改,所以不能加标题。那重新来: 首先,原文的内容,分成几个p?比如:

Depth recurrence increases effective model depth by repeatedly applying a fixed computation block. However, sharing attention parameters does not share the key-value (KV) states generated at different recurrence iterations, which causes memory growth with recurrence depth during autoregressive decoding. We study how to reduce this memory cost through architectural and cache-sharing approaches within a common depth-recurrent framework.

循环混合架构研究:?不对,不能加内容。哦对,不能增删,所以不能加任何原文没有的内容,包括小节标题。那h2/h3只能用原文里的内容?但原文里没有,那是不是不用h2/h3?那直接把原文的内容拆成几个p?不对,原文是一整段,但可以按照句子组拆分?不对,规则说“每个自然段落用

包裹”,那原文的自然段落是一个,但如果内容长,可以拆分?不对,再仔细看用户给的正文,其实是一整段,但逻辑上有四个部分:背景、三个方法、结论。那如果不用h2/h3,就用p,每个逻辑部分一个p?但要注意不能增删内容。那:

Depth recurrence increases effective model depth by repeatedly applying a fixed computation block. However, sharing attention parameters does not share the key-value (KV) states generated at different recurrence iterations, which causes memory growth with recurrence depth during autoregressive decoding. We study how to reduce this memory cost through architectural and cache-sharing approaches within a common depth-recurrent framework.

We first study a looped hybrid that repeats a stack of Transformer layers, with recurrent mLSTM or Gated DeltaNet blocks replacing most attention layers and periodic Transformer attention anchors retained.

We then restrict recurrence to a dedicated recurrent workspace in a Parcae-style architecture and test its stabilising adapter as a wrapper for the looped hybrid.

Finally, we train a looped Transformer with recurrent KV replay, where only selected recurrence iterations write KV states that later iterations reuse. This trains the model to operate with the same shared KV topology used during inference.

We find that looped hybrids require sufficiently frequent attention anchors, with performance depending strongly on the recurrent mixer. For cross-iteration KV sharing, recurrent KV replay allows multiple recurrence iterations to reuse the same KV states while retaining performance close to the full looped Transformer with separate KV caches for each iteration.

但这样是不是符合要求?或者有没有需要用strong的地方?比如重要句子,比如“We study how to reduce this memory cost through architectural and cache-sharing approaches within a common depth-recurrent framework.”可以用strong?还有最后结论的部分可以用strong?但规则说重要句子可用strong,不是必须。那再调整,比如:

Depth recurrence increases effective model depth by repeatedly applying a fixed computation block. However, sharing attention parameters does not share the key-value (KV) states generated at different recurrence iterations, which causes memory growth with recurrence depth during autoregressive decoding. We study how to reduce this memory cost through architectural and cache-sharing approaches within a common depth-recurrent framework.

We first study a looped hybrid that repeats a stack of Transformer layers, with recurrent mLSTM or Gated DeltaNet blocks replacing most attention layers and periodic Transformer attention anchors retained.

We then restrict recurrence to a dedicated recurrent workspace in a Parcae-style architecture and test its stabilising adapter as a wrapper for the looped hybrid.

Finally, we train a looped Transformer with recurrent KV replay, where only