首页 > AI前沿 > On-Policy Morphing into Hybrid Attention Models

On-Policy Morphing into Hybrid Attention Models

arXiv自然语言 2026-11-19 01:45 31 阅读 查看原文
用户需要将一段纯文本新闻(实际上是学术论文摘要)排版成语义化的HTML片段。 规则要求: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/``等文档级标签。
    
    分析原文:
    "Hybrid attention models combine full and linear attention to achieve a better quality--efficiency trade-off. Converting pretrained Transformers into such models requires identifying which layers should retain full attention under a prescribed budget. We identify two off-policy mismatches in FlashMorph's layer selection: (i) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; and (ii) frozen model weights fail to account for parameter adaptation during subsequent recovery. We introduce FlashMorph2, an on-policy layer-selection method that (I) evaluates sampled discrete hybrid architectures end to end using their own hidden states, and (II) jointly optimizes layer-selection scores and model weights under the recovery objective to reflect post-adaptation performance. Experiments across Qwen3 model scales and linear-attention variants show that FlashMorph2 consistently outperforms existing layer-selection methods, improving average scores over FlashMorph by $+2.42$ points on RULER, $+1.08$ on LongBench, and $+0.76$ on recall-intensive tasks, while completing layer selection in only $0.72$--$3.57$ GPU hours across 0.6B--8B models, demonstrating high efficiency and scalability."
    
    这段文本虽然是一个段落,但可以划分为几个逻辑小节,或者就保持一个段落。因为原文是一整段,没有明显的列表或引用、代码。
    不过为了应用一些标签,我可以把它分成几个部分,或者就简单地用一个`

    `,里面用``强调重要句子。 原文中有两个列出的点: "(i) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; and (ii) frozen model weights fail to account for parameter adaptation during subsequent recovery." 还有: "(I) evaluates sampled discrete hybrid architectures end to end using their own hidden states, and (II) jointly optimizes layer-selection scores and model weights under the recovery objective to reflect post-adaptation performance." 我可以把这两处提取为`

    • `,这样更符合规则3。 但是要注意,不能增删改内容。提取为列表时,需要把原来的连接词处理好,或者保留在`
    • `里。 比如: "We identify two off-policy mismatches in FlashMorph's layer selection:
      • (i) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; and
      • (ii) frozen model weights fail to account for parameter adaptation during subsequent recovery.
      " 或者: "We identify two off-policy mismatches in FlashMorph's layer selection:
      • (i) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; and
      • (ii) frozen model weights fail to account for parameter adaptation during subsequent recovery.
      " 同理: "We introduce FlashMorph2, an on-policy layer-selection method that
      • (I) evaluates sampled discrete hybrid architectures end to end using their own hidden states, and
      • (II) jointly optimizes layer-selection scores and model weights under the recovery objective to reflect post-adaptation performance.
      " 这样改动了原文的标点(把逗号或句号放到列表里),算不算增删改? 为了严格保留原文所有信息,不改变文字,我可以直接把整段放在一个`

      `中,或者分成几个`

      `。 其实规则说“每个自然段落用`

      `包裹”,原文就是一个自然段落。 "长文用`

      `/`

      `划分小节" 这段不长,可以不加h2/h3,或者加一个h2作为标题?原文没有标题,不要自己加,以免增删内容。 所以,直接用一个`

      `包裹整个段落,里面使用``来强调重要句子。 重要句子: "We introduce FlashMorph2, an on-policy layer-selection method..." 可以加粗。 "Experiments across Qwen3 model scales..." 可以加粗。 让我们仔细看: `

      Hybrid attention models combine full and linear attention to achieve a better quality--efficiency trade-off. Converting pretrained Transformers into such models requires identifying which layers should retain full attention under a prescribed budget. We identify two off-policy mismatches in FlashMorph's layer selection: (i) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; and (ii) frozen model weights fail to account for parameter adaptation during subsequent recovery. We introduce FlashMorph2, an on-policy layer-selection method that (I) evaluates sampled discrete hybrid architectures end to end using their own hidden states, and (II) jointly optimizes layer-selection scores and model weights under the recovery objective to reflect post-adaptation performance. Experiments across Qwen3 model scales and linear-attention variants show that FlashMorph2 consistently outperforms existing layer-selection methods, improving average scores over FlashMorph by $+2.42$ points on RULER, $+1.08$ on LongBench, and $+0.76$ on recall-intensive tasks, while completing layer selection in only $0.72$--$3.57$ GPU hours across 0.6B--8B models, demonstrating high efficiency and scalability.

      ` 检查是否满足所有规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 -> 是 2. 每个自然段落用`

      `包裹,长文用`

      `/`

      `划分小节。 -> 原文是一段,用`

      `包裹。 3. 列表内容用`

      • `,引用用`
        `,代码用`
        `,重要句子可用``。 -> 使用了``。没有列表、引用、代码,所以不用强行加,否则会增改内容。
        4. 保留原文所有信息,不得增删改内容。 -> 完全保留。
        5. 不要输出``/``/``等文档级标签。 -> 是。
        
        有没有可能把(i)和(ii)做成列表?
        "We identify two off-policy mismatches in FlashMorph's layer selection: (i) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; and (ii) frozen model weights fail to account for parameter adaptation during subsequent recovery."
        如果做成列表:
        We identify two off-policy mismatches in FlashMorph's layer selection:
        
        • (i) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; and
        • (ii) frozen model weights fail to account for parameter adaptation during subsequent recovery.
        这会改变排版结构,但文本内容没变。不过为了最安全地“不增删改内容”,保持纯文本流加上``是最稳妥的。因为原文本就是内联的(i)和(ii),并没有换行。 再审视一下文本,是否有需要转义的字符? 没有`<`、`>`、`&`。`$`符号不需要转义。 最终输出:

        Hybrid attention models combine full and linear attention to achieve a better quality--efficiency trade-off. Converting pretrained Transformers into such models requires identifying which layers should retain full attention under a prescribed budget. We identify two off-policy mismatches in FlashMorph's layer selection: (i) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; and (ii) frozen model weights fail to account for parameter adaptation during subsequent recovery. We introduce FlashMorph2, an on-policy layer-selection method that (I) evaluates sampled discrete hybrid architectures end to end using their