首页 > AI前沿 > POEM: Output-Relative Block-Sparse Attention for Long-Context Prefilling

POEM: Output-Relative Block-Sparse Attention for Long-Context Prefilling

arXiv自然语言 2026-11-10 12:52 12 阅读 查看原文
用户现在让我把给定的新闻正文排版成语义化的HTML片段,首先看正文结构,开头是一个长自然段,里面讲LLM的能力、Jailbreak攻击、现有防御的问题、提出TERM方法、相关的技术细节还有实验对比。首先按照规则,每个自然段落用

包裹,不过这里正文看起来是一个连续的段落,但需要检查有没有可以划分的小节?不对,原文是一整段,但看内容,前半部分讲背景和方法提出,中间讲技术调整,最后讲实验和结果。不过先看规则,长文用

/

划分小节,那得先看内容逻辑。首先,开头第一句到“induce LLMs to generate harmful responses.”是背景,然后接下来到“limiting over-refusal remains a challenge.”是现有防御的挑战,然后从“In this paper, we propose Target-free Entropy-guided Residual Modulation (TERM)”到“without relying on a predefined safety direction.”是方法的提出和技术细节,最后“We compare our method with seven defenses”到最后是实验结果。那这样划分的话,需要给每个小节加标题吗?不对,原文没有标题,规则是长文用

/

划分小节,但要保留原文所有信息,不得增删改内容,所以标题应该是原文里的内容吗?不对,原文是一段,那可能直接用

?不对,再仔细看规则:2.每个自然段落用

包裹,长文用

/

划分小节。那原文虽然是一段,但内容有逻辑分段,比如背景部分、挑战部分、方法部分、实验部分,那可以用

来划分这些小节的标题吗?但原文没有标题,那是不是不能加标题?不对,规则说“长文用

/

划分小节”,但不得增删改内容,所以标题应该是原文中的句子作为小节的引导?不对,再看原文,其实原文是一个连贯的段落,可能直接用

包裹整个?不对,再看规则4:保留原文所有信息,不得增删改内容。那原文的所有句子都要包含,不能拆分标题。那再仔细看,原文的内容:首先讲LLM的能力、Jailbreak攻击,然后现有防御的问题,然后提出TERM方法,然后讲方法的技术细节(attention entropy的校准、自适应缩放等),然后实验和结果。那这样的话,可能直接把整个正文用

包裹?不对,规则2说长文用

/

划分小节,那可能需要把原文的内容按逻辑分成小节,每个小节用

或

,然后内容用

。比如:

Background and Challenges

但原文没有这个标题,所以不行,因为不能增删内容。哦,不对,规则说“不得增删改内容”,所以不能添加原文没有的标题。那怎么办?那原文是一个完整的段落,所以直接用

包裹整个内容?但再看规则2,长文用

/

划分小节,那可能原文里的内容有可以作为小节主题的句子,比如“In this paper, we propose Target-free Entropy-guided Residual Modulation (TERM), an inference-time defense that uses prompt-calibrated attention entropy deviations to guide adaptive modulation of the model's own residual updates.”这句话是方法的提出,那可以用

包裹这句话?不对,标题应该是概括性的,但原文没有标题,那可能只能把整个正文用

包裹?不对,再检查原文的结构: 原文句子顺序: 1. Large language models (LLMs) have shown strong capabilities across diverse applications, making both safety and utility essential for reliable deployment. 2. Jailbreak attacks use diverse forms of adversarial prompting to bypass alignment safeguards and induce LLMs to generate harmful responses. 3. Some representation-based defenses regulate internal activations using predefined steering directions or learned safety regions. 4. However, adapting intervention to each input without a predefined representational target while limiting over-refusal remains a challenge. 5. In this paper, we propose Target-free Entropy-guided Residual Modulation (TERM), an inference-time defense that uses prompt-calibrated attention entropy deviations to guide adaptive modulation of the model's own residual updates. 6. Attention patterns vary across prompts and decoding steps, making raw entropy values difficult to interpret as a consistent signal for intervention. 7. We therefore calibrate attention entropy against each prompt's own prefill baseline to establish an input-specific reference for reducing unnecessary intervention on benign requests. 8. Adaptive scale normalization and temporal accumulation then account for differences in deviation scale and persistence to determine intervention strength. 9. The resulting signal controls bounded scaling of the model's own residual updates, without relying on a predefined safety direction. 10. We compare our method with seven defenses on seven jailbreak attacks and two over-refusal benchmarks across Llama, Qwen, and Gemma models, including Qwen3 models from 0.6B to 32B. 11. Results show our method improves jailbreak robustness while maintaining low over-refusal on benign safety-sensitive requests with modest inference overhead. 那这些句子里,第5句是方法提出的开始,第10句是实验部分的开始,那可以用

包裹第5句作为方法部分的标题?不对,标题应该是短语,不是完整句子?或者直接把每个逻辑部分用

,然后用强调重要句子,比如方法名称、结果。比如:

Large language models (LLMs) have shown strong capabilities across diverse applications, making both safety and utility essential for reliable deployment. Jailbreak attacks use diverse forms of adversarial prompting to bypass alignment safeguards and induce LLMs to generate harmful responses. Some representation-based defenses regulate internal activations using predefined steering directions or learned safety regions. However, adapting intervention to each input without a predefined representational target while limiting over-refusal remains a challenge.

In this paper, we propose Target-free Entropy-guided Residual Modulation (TERM), an inference-time defense that uses prompt-calibrated attention entropy deviations to guide adaptive modulation of the model's own residual updates. Attention patterns vary across prompts and decoding steps, making raw entropy values difficult to interpret as a consistent signal for intervention. We therefore calibrate attention entropy against each prompt's own prefill baseline to establish an input-specific reference for reducing unnecessary intervention on benign requests. Adaptive scale normalization and temporal accumulation then account for differences in deviation scale and persistence to determine intervention strength. The resulting signal controls bounded scaling of the model's own residual updates, without relying on a predefined safety direction.

We compare our method with seven defenses on seven jailbreak attacks and two over-refusal benchmarks across Llama, Qwen, and Gemma models, including Qwen3 models from 0.6B to 32B. Results show our method improves jailbreak robustness while maintaining low over-refusal on benign safety-sensitive requests with modest inference overhead.

这样对吗?因为原文的内容被分成了三个自然段?不对,原文是一个自然段,但用户给的正文是一段,那是不是可以按逻辑分成三个

?规则2说每个自然段落用

包裹,那原文是一个自然段落,所以应该用一个

?但这样太长,规则2说长文用

/

划分小节,那可能需要用

来划分小节,比如:

Background and Existing Challenges

但不行,因为原文没有这个标题,不能添加。哦,不对,规则说“保留原文所有信息,不得增删改内容”,所以不能添加任何原文没有的内容,包括标题。那怎么办?那原文的所有内容都是连贯的,所以直接用

包裹整个内容?然后重要的句子用,比如方法名称、结果。比如:

Large language models (LLMs) have shown strong capabilities across diverse applications, making both safety and utility essential for reliable deployment. Jailbreak attacks use diverse forms of adversarial prompting to bypass alignment safeguards and induce LLMs to generate harmful responses. Some representation-based defenses regulate internal activations using predefined steering directions or learned safety regions. However, adapting intervention to each input without a predefined representational target while limiting over-refusal remains a challenge. In this paper, we propose Target-free Entropy-guided Residual Modulation (TERM), an inference-time defense that uses prompt-calibrated attention entropy deviations to guide adaptive modulation of the model's own residual updates. Attention patterns vary across prompts and decoding steps, making raw