`包裹,长文用`
`/`
`划分小节。 3. 列表内容用`
- `,引用用`
`,代码用`
`,重要句子可用``。
4. 保留原文所有信息,不得增删改内容。
5. 不要输出``/``/``等文档级标签。
原文:
Recurrent memories let language models process long sequences with a fixed-size state. Linear memories such as DeltaNet and KDA store one matrix and take one delta-rule step per token. The product of a chunk of such steps has an exact WY form, so they train fast. Deep memories such as TTT-MLP and Titans store a small MLP, which can hold maps that no matrix can, but their update has no WY form. Toaster bridges the two by building on a known fact: a gradient step on a linear layer is a delta-rule write. It builds a two-layer nonlinear memory from two linear memories with an activation between them. We use target propagation to give the hidden memory its values by projecting the output error back through the second memory. Given a chunk's cross-layer signals, approximated once per chunk, each layer can then run on an unmodified delta-rule kernel. Our default runs this kernel on the output layer and updates the hidden layer once per chunk. Running it on both layers gives a better model in our ablation. At matched backbones and training-token budgets, Toaster has lower loss than KDA at all five tested sizes (0.2B-1.6B non-embedding parameters) on both C4 and PG19. Its C4 perplexity also matches or slightly improves on the attention baseline at every size. At 1.2B it is also below Titans. In a 3:1 hybrid with global attention, the Toaster hybrid has the lowest perplexity of the four hybrids tested, and it is 0.18 (C4) and 0.44 (PG19) below the attention baseline. At the largest size, Toaster's mean three-shot accuracy on 13 tasks is 51.84%, compared with 50.52% for KDA and 50.04% for Titans. In length extrapolation, Toaster's loss stays below KDA's up to 20K tokens (5x the training length). On the 145M backbone, at a hidden width matched to Titans' memory, Toaster takes at most 1.33x KDA's training-step time and is 2.8-5.5x faster than the tested Titans configurations at 32K context.
分析原文结构:
原文是一整段,没有明显的小节标题。为了符合“长文用``/`
`划分小节”的要求,我可以将其逻辑划分为几个小节,例如:
- 背景与现有方法 (Linear memories vs Deep memories)
- Toaster 的架构与方法 (Toaster's Approach)
- 实验结果与性能 (Performance and Results)
但是规则要求“保留原文所有信息,不得增删改内容”。如果我添加小标题,算不算“增改内容”?
“不得增删改内容”通常指正文的文字不能改,但添加结构性的HTML标签(如h2/h3)是排版要求。不过,如果我在h2/h3里写了文字,就相当于增加了原文没有的文字。
为了严格遵守“不得增删改内容”,我最好不添加任何原文没有的文字作为标题,或者只用纯粹的HTML标签划分,但h2/h3必须有内容。
或者,我可以把原文的某几句话提炼为小标题?不,这属于修改/增加。
等等,规则说“长文用
/
划分小节”,如果原文没有标题,我是否可以不加h2/h3,只用p?“长文用...划分小节”,如果我觉得不够长或者不需要划分,可以不用。但为了体现排版能力,我可以把原文的几个逻辑块用p包裹,如果非要用h2/h3,可能会引入额外文字。
或者,我可以把原文的第一句话或某个核心句作为h3?不行,这改变了原文结构。
其实,如果不加h2/h3,只使用`
`也是符合“每个自然段落用`
`包裹”的。原文只有一个自然段落,那我就只用一个`
`?
但原文其实可以拆分成几个自然段,以符合阅读习惯。比如:
段1:Recurrent memories... no WY form.
段2:Toaster bridges the two... better model in our ablation.
段3:At matched backbones... 32K context.
这样拆分自然段,不增加任何文字,完全保留原文内容。
再考虑是否需要h2/h3。如果不用h2/h3,只输出几个`
`,是否违反规则2?“长文用
`,代码用``,重要句子可用``。 4. 保留原文所有信息,不得增删改内容。 5. 不要输出``/``/``等文档级标签。 原文: Recurrent memories let language models process long sequences with a fixed-size state. Linear memories such as DeltaNet and KDA store one matrix and take one delta-rule step per token. The product of a chunk of such steps has an exact WY form, so they train fast. Deep memories such as TTT-MLP and Titans store a small MLP, which can hold maps that no matrix can, but their update has no WY form. Toaster bridges the two by building on a known fact: a gradient step on a linear layer is a delta-rule write. It builds a two-layer nonlinear memory from two linear memories with an activation between them. We use target propagation to give the hidden memory its values by projecting the output error back through the second memory. Given a chunk's cross-layer signals, approximated once per chunk, each layer can then run on an unmodified delta-rule kernel. Our default runs this kernel on the output layer and updates the hidden layer once per chunk. Running it on both layers gives a better model in our ablation. At matched backbones and training-token budgets, Toaster has lower loss than KDA at all five tested sizes (0.2B-1.6B non-embedding parameters) on both C4 and PG19. Its C4 perplexity also matches or slightly improves on the attention baseline at every size. At 1.2B it is also below Titans. In a 3:1 hybrid with global attention, the Toaster hybrid has the lowest perplexity of the four hybrids tested, and it is 0.18 (C4) and 0.44 (PG19) below the attention baseline. At the largest size, Toaster's mean three-shot accuracy on 13 tasks is 51.84%, compared with 50.52% for KDA and 50.04% for Titans. In length extrapolation, Toaster's loss stays below KDA's up to 20K tokens (5x the training length). On the 145M backbone, at a hidden width matched to Titans' memory, Toaster takes at most 1.33x KDA's training-step time and is 2.8-5.5x faster than the tested Titans configurations at 32K context. 分析原文结构: 原文是一整段,没有明显的小节标题。为了符合“长文用``/`
`划分小节”的要求,我可以将其逻辑划分为几个小节,例如: - 背景与现有方法 (Linear memories vs Deep memories) - Toaster 的架构与方法 (Toaster's Approach) - 实验结果与性能 (Performance and Results) 但是规则要求“保留原文所有信息,不得增删改内容”。如果我添加小标题,算不算“增改内容”? “不得增删改内容”通常指正文的文字不能改,但添加结构性的HTML标签(如h2/h3)是排版要求。不过,如果我在h2/h3里写了文字,就相当于增加了原文没有的文字。 为了严格遵守“不得增删改内容”,我最好不添加任何原文没有的文字作为标题,或者只用纯粹的HTML标签划分,但h2/h3必须有内容。 或者,我可以把原文的某几句话提炼为小标题?不,这属于修改/增加。 等等,规则说“长文用
/
划分小节”,如果原文没有标题,我是否可以不加h2/h3,只用p?“长文用...划分小节”,如果我觉得不够长或者不需要划分,可以不用。但为了体现排版能力,我可以把原文的几个逻辑块用p包裹,如果非要用h2/h3,可能会引入额外文字。 或者,我可以把原文的第一句话或某个核心句作为h3?不行,这改变了原文结构。 其实,如果不加h2/h3,只使用`
`也是符合“每个自然段落用`
`包裹”的。原文只有一个自然段落,那我就只用一个`
`? 但原文其实可以拆分成几个自然段,以符合阅读习惯。比如: 段1:Recurrent memories... no WY form. 段2:Toaster bridges the two... better model in our ablation. 段3:At matched backbones... 32K context. 这样拆分自然段,不增加任何文字,完全保留原文内容。 再考虑是否需要h2/h3。如果不用h2/h3,只输出几个`
`,是否违反规则2?“长文用