首页 > AI前沿 > FrontierReplay: Split-Free Exact Prefix Caching for Hybrid Language Models

FrontierReplay: Split-Free Exact Prefix Caching for Hybrid Language Models

arXiv机器学习 2026-11-19 20:34 12 阅读 查看原文
用户现在需要把给定的英文新闻正文排版成语义化的HTML片段,得遵循给定的规则。首先,先看内容,整个是一段介绍FrontierReplay的,没有分小节的话?不对,看内容,开头是背景,然后是观察,然后是提出的系统,然后是评估结果对吧?不对,原文是连续的自然段吗?不对,原文的内容是几个部分:首先讲混合语言模型的问题,然后是观察到的点,然后介绍FrontierReplay,然后讲它的工作方式,然后讲评估结果和对比。首先,每个自然段落用p包裹,有没有需要h2/h3的?看内容,如果可以划分的话,比如第一个部分是背景问题,第二个是观察,第三个是提出系统和机制,第四个是评估结果?不对,不对,原文是连续的几段吗?不对,先看原文的句群: 第一部分:Hybrid language models... under load. 这部分是背景和现有问题。 然后第二部分:We observe that... known branch points. 这是观察到的现象。 第三部分:We present FrontierReplay... without host synchronization. 这是提出系统和核心机制。 第四部分:A frontier-aware retention policy... attention KV cache. 这是保留策略。 第五部分:Across diverse hybrid architectures... long contexts. 这是评估结果和对比。 不对,但是要不要加小标题?哦,规则说长文用h2/h3划分小节,那可以合理划分吗?不对,不能增删改内容,小标题是要加吗?不对,规则是保留原文所有信息,不得增删改内容,哦,不对,小标题如果是原文没有的是不是不行?哦,不对,再看规则:3. 长文用h2/h3划分小节,但是不能增删改内容,那是不是如果没有原文的小标题的话,要不要加?不对,再仔细看原文,原文是连续的论述,没有小标题,那是不是直接分段落?不对,再看,原文的内容: 首先,第一句到under load是讲现有混合模型的prefix复用的问题,这是一个部分。然后We observe that...是观察结果。然后We present FrontierReplay...是介绍系统。然后A frontier-aware...是策略。然后Across diverse...是结果。 不对,但是要不要加h2?如果加的话,小标题不能自己造,因为不能增内容?哦,对,规则4说保留原文所有信息,不得增删改内容,所以不能自己加小标题啊!哦,对,那h2/h3是划分小节,但是原文没有小节标题的话,是不是就不用加?不对,再看规则,长文用h2/h3划分小节,但是不能增内容,那是不是原文的内容里有没有可以作为小节的?不对,原文是纯文本,没有小标题,那可能直接分p?不对,再仔细看原文的结构: 第一段(第一个自然段):Hybrid language models combine self-attention with recurrent layers whose in-place state updates restrict prefix reuse to exact saved positions. Saving states on a coarse grid wastes reusable prefixes, whereas existing fine-grained exact caches must cut the prefill forward pass at each mid-prompt checkpoint, adding scheduler steps that outweigh the reuse gains under load. 这是一个背景和现有问题的段落。 第二段(第二个自然段):We observe that across multi-turn chat, agent loops, document question answering, and tree search, prefix reuse almost always ends at structural boundaries predictable in advance, such as turn ends and known branch points. 这是观察结果的段落。 第三段(第三个自然段):We present FrontierReplay, a prefix caching system that materializes exact checkpoints at these boundaries without cutting the prefill. While the forward pass runs uncut, FrontierReplay logs a short window of recurrent-layer inputs and invokes a cross-layer, state-only replay kernel to build the target state 6-37x cheaper per token than full recomputation without host synchronization. 这是提出系统和核心机制的段落,这里可以把重要的句子用strong?比如We present FrontierReplay这个句子,还有核心机制的部分? 第四段(第四个自然段):A frontier-aware retention policy prioritizes active anchors and proven branches while lending dead checkpoint slots to the attention KV cache. 这是策略的段落。 第五段(第五个自然段):Across diverse hybrid architectures, model scales, and workloads, FrontierReplay delivers clear gains over existing exact caches, lowering median time to first token (TTFT) by 16-89% (up to 8.9x), raising 80% service-level objective (SLO) capacity by up to 1.73x, and improving goodput by up to 4.26x (3.67x at larger scales). Compared with approximate replay methods, FrontierReplay delivers up to 1.54x higher goodput while preserving exact task accuracy, avoiding the up to 17% accuracy drop suffered by approximation on long contexts. 这是评估结果和对比的段落,这里的性能数据可以用strong? 等下,但是有没有需要列表或者引用或者代码?原文里没有,所以不用。那现在,要不要加h2/h3?因为如果加的话,必须用原文的内容?不对,原文没有小标题,所以不能自己加,因为不能增删改内容。哦,对,规则4说不得增删改内容,所以不能自己造小标题。那是不是就分p,然后重要的句子用strong? 等下,再仔细看规则2:长文用

/

划分小节。那如果原文没有小节的话,是不是就不用?不对,再想想,原文的内容结构,是不是可以不用h2,因为都是连贯的论述,没有分节的原文提示。那现在组织HTML: 首先第一个p:内容是现有问题,有没有重要句子?比如第一句?不对,看,重要的点,比如"existing fine-grained exact caches must cut the prefill forward pass at each mid-prompt checkpoint, adding scheduler steps that outweigh the reuse gains under load" 可以加strong? 然后第二个p:We observe that...部分,是不是"prefix reuse almost always ends at structural boundaries predictable in advance"可以加strong? 第三个p:We present FrontierReplay...这里,"We present FrontierReplay"可以加strong,还有后面的"invokes a cross-layer, state-only replay kernel to build the target state 6-37x cheaper per token than full recomputation without host synchronization"可以加strong? 第四个p:A frontier-aware retention policy...这个可以整句?或者不用? 第五个p:Across diverse...这里,"lowering median time to first token (TTFT) by 16-89% (up to 8.9x), raising 80% service-level objective (SLO) capacity by up to 1.73x, and improving goodput by up to 4.26x (3.67x at larger scales)"和"delivers up to 1.54x higher goodput while preserving exact task accuracy, avoiding the up to 17% accuracy drop suffered by approximation on long contexts"可以加strong? 等下,但是要注意,不能增删改内容,所以strong只是标记,不改变内容。那现在写出来:

Hybrid language models combine self-attention with recurrent layers whose in-place state updates restrict prefix reuse to exact saved positions. Saving states on a coarse grid wastes reusable prefixes, whereas existing fine-grained exact caches must cut the prefill forward pass at each mid-prompt checkpoint, adding scheduler steps that outweigh the reuse gains under load.

We observe that across multi-turn chat, agent loops, document question answering, and tree search, prefix reuse almost always ends at structural boundaries predictable in advance, such as turn ends and known branch points.

We present FrontierReplay, a prefix caching system that materializes exact checkpoints at these boundaries without cutting the prefill. While the forward pass runs uncut, FrontierReplay logs a short window of recurrent-layer inputs and invokes a cross-layer, state-only replay kernel to build the target state 6-37x cheaper per token than full recomputation without host synchronization.

A frontier-aware retention policy prioritizes active anchors and proven branches while lending dead checkpoint slots to the attention KV cache.

Across diverse hybrid architectures, model scales, and workloads, FrontierReplay delivers clear gains over existing exact caches, lowering median time to first token (TTFT) by 16-89% (up to 8.9x), raising 80% service-level objective (SLO) capacity by up to 1.73x, and improving goodput by up