首页 > AI前沿 > Mask-Pattern Position Augmentation: Training-Free Long-Context Extrapolation for Diffusion LLMs

Mask-Pattern Position Augmentation: Training-Free Long-Context Extrapolation for Diffusion LLMs

arXiv机器学习 2026-11-11 12:48 19 阅读 查看原文
用户现在需要把给定的英文新闻正文排版成语义化的HTML片段,按照规则来。首先看正文的结构,首先有几个自然段对吧?首先第一段是讲dLLM的问题,然后第二段讲现有方法的不足,第三段提出MPPA,第四段讲MPPA的原理,第五段讲实验结果,最后一句是结论?不对,先拆分自然段落: 首先第一句是第一个段落?不对,原文的段落:哦原文的纯文本是一整段?不对看原文: 第一句:Diffusion large language models (dLLMs) generate by iteratively unmasking tokens with bidirectional attention, but rotary position embeddings (RoPE) still confine them to a fixed pretraining window. 不对?不对再看,哦不对,原文的内容是: 1. 第一部分:Diffusion large language models (dLLMs) generate by iteratively unmasking tokens with bidirectional attention, but rotary position embeddings (RoPE) still confine them to a fixed pretraining window. 不对?不对,下一句是Existing training-free extrapolation, notably LongLLaDA, copies NTK-based RoPE scaling from autoregressive (AR) models and applies one static, length-only transform at every position and denoising step. 然后是That transplant ignores a signal unique to dLLMs: the evolving mask pattern of the partially decoded sequence. 然后是We propose Mask-Pattern Position Augmentation (MPPA). 然后是MPPA linearly interpolates indices into the pretrained window, then adds a per-position, per-step offset from cumulative mask density. 然后是The offset restores rotary phase resolution that interpolation collapses, and it shrinks as tokens are committed, so late steps see near-clean interpolated geometry. 然后是On LLaDA-8B-Instruct at twice its 4096-token window, MPPA reduces perplexity from 19.51 under standard decoding to 3.12 at 8192 tokens, below the in-window figure, with 100% passkey retrieval and unchanged needle-in-a-haystack accuracy. 然后是These preliminary $2\times$ results motivate mask-conditioned position engineering as a diffusion-native path to long-context generation. 不对,这些是自然段落吗?不对,用户说每个自然段落用

包裹,长文用h2h3划分小节。哦不对,先看内容的逻辑:首先背景介绍dLLM的局限,然后现有方法的问题,然后提出方法MPPA,然后方法原理,然后实验结果,最后结论。那可以划分小节吗?不对,有没有需要用h2/h3的?哦看内容,比如: 首先

背景与现有方法局限

?不对不对,规则说不要增删改内容哦!哦对!规则4是保留原文所有信息,不得增删改内容。哦对,所以不能自己加标题文字,因为原文没有标题。哦那h2/h3是用来划分原文里的小节?不对原文里没有小标题啊。哦不对原文的内容是连续的,那自然段落的话,原文的纯文本里的换行?不对用户给的正文是连续的一段?不对看用户给的正文: 正文如下: Diffusion large language models (dLLMs) generate by iteratively unmasking tokens with bidirectional attention, but rotary position embeddings (RoPE) still confine them to a fixed pretraining window. Existing training-free extrapolation, notably LongLLaDA, copies NTK-based RoPE scaling from autoregressive (AR) models and applies one static, length-only transform at every position and denoising step. That transplant ignores a signal unique to dLLMs: the evolving mask pattern of the partially decoded sequence. We propose Mask-Pattern Position Augmentation (MPPA). MPPA linearly interpolates indices into the pretrained window, then adds a per-position, per-step offset from cumulative mask density. The offset restores rotary phase resolution that interpolation collapses, and it shrinks as tokens are committed, so late steps see near-clean interpolated geometry. On LLaDA-8B-Instruct at twice its 4096-token window, MPPA reduces perplexity from 19.51 under standard decoding to 3.12 at 8192 tokens, below the in-window figure, with 100% passkey retrieval and unchanged needle-in-a-haystack accuracy. These preliminary $2\times$ results motivate mask-conditioned position engineering as a diffusion-native path to long-context generation. 哦原来整篇是一个自然段落?不对不对,自然段落是按原文的换行,但是用户给的正文是一整段没有换行?那不对,那怎么处理?哦不对,可能用户给的正文的内容的逻辑段落,我们按内容拆分?不对规则说不得增删改内容,拆分段落不算改内容对吧?对,因为只是排版,内容不变。那首先,有没有需要用h2/h3的?哦原文里没有小标题,所以不需要用h2h3?不对,规则说“长文用

/

划分小节”,如果原文本身没有小节内容的话,就不用加,因为不能自己加文字。对,因为不能增删改内容,所以标题不能自己造。那接下来,每个自然段落用p,那原文的内容可以按逻辑拆分几个p?比如: 第一个p:Diffusion large language models (dLLMs) generate by iteratively unmasking tokens with bidirectional attention, but rotary position embeddings (RoPE) still confine them to a fixed pretraining window. Existing training-free extrapolation, notably LongLLaDA, copies NTK-based RoPE scaling from autoregressive (AR) models and applies one static, length-only transform at every position and denoising step. That transplant ignores a signal unique to dLLMs: the evolving mask pattern of the partially decoded sequence. 然后第二个p:We propose Mask-Pattern Position Augmentation (MPPA). 不对,重要的句子可以用strong,比如提出方法的句子是重要的?然后第三个p:MPPA linearly interpolates indices into the pretrained window, then adds a per-position, per-step offset from cumulative mask density. The offset restores rotary phase resolution that interpolation collapses, and it shrinks as tokens are committed, so late steps see near-clean interpolated geometry. 然后第四个p:On LLaDA-8B-Instruct at twice its 4096-token window, MPPA reduces perplexity from 19.51 under standard decoding to 3.12 at 8192 tokens, below the in-window figure, with 100% passkey retrieval and unchanged needle-in-a-haystack accuracy. 然后第五个p:These preliminary $2\times$ results motivate mask-conditioned position engineering as a diffusion-native path to long-context generation. 不对,那有没有需要列表?没有,原文没有列表。引用?没有。代码?没有。那检查有没有重要句子需要加strong,比如提出MPPA的句子,还有最后结论的句子,还有那个效果的句子?比如“MPPA reduces perplexity from 19.51 under standard decoding to 3.12 at 8192 tokens, below the in-window figure, with 100% passkey retrieval and unchanged needle-in-a-haystack accuracy.”这个也可以加strong? 等下,不要改内容,只是加标签。那现在组织: 首先,第一段内容:Diffusion large language models (dLLMs) generate by iteratively unmasking tokens with bidirectional attention, but rotary position embeddings (RoPE) still confine them to a fixed pretraining window. Existing training-free extrapolation, notably LongLLaDA, copies NTK-based RoPE scaling from autoregressive (AR) models and applies one static, length-only transform at every position and denoising step. That transplant ignores a signal unique to dLLMs: the evolving mask pattern of the partially decoded sequence. 用

包。 然后第二段:

We propose Mask-Pattern Position Augmentation (MPPA).

然后第三段:

MPPA linearly interpolates indices into the pretrained window, then adds a per-position, per-step offset from cumulative mask density. The offset restores rotary phase resolution that interpolation collapses, and it shrinks as tokens are committed, so late steps see near-clean interpolated geometry.

然后第四段:

On LLaDA-8B-Instruct at twice its 4096-token window, MPPA reduces perplexity from