首页 > AI前沿 > BOS Summary: The First Token Matters

BOS Summary: The First Token Matters

arXiv自然语言 2026-11-18 05:01 25 阅读 查看原文
用户需要我作为一个专业的新闻排版工程师,将提供的一段纯文本新闻正文排版成语义化的HTML片段。 需要遵循以下规则: 1. 只输出HTML片段本身,不要Markdown围栏(如```html),不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/``等文档级标签。
    
    分析提供的正文:
    "Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly access carries no information about the input. We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context. At a single transformer layer during prefill, BOS Summary relaxes the causal constraint for the BOS query row alone, letting BOS aggregate all visual and textual input tokens into a self-derived context representation while every other token retains standard causal attention. The enriched representation then remains accessible to the model throughout decoding, with no additional parameters, auxiliary modules, or decoding-time overhead. Across multiple VLM and LLM families and eight multimodal and text-only benchmarks, BOS Summary improves over direct and Chain-of-Thought prompting and outperforms recent sink-intervention methods on multi-step reasoning tasks. A mechanistic analysis shows that making BOS input-specific also weakens its sink dominance and redistributes attention toward content-bearing prompt tokens, indicating that changing what a sink carries reshapes how later layers allocate attention."
    
    这是一段连续的学术新闻/摘要,没有明显的列表、引用或代码。可以将其分为几个自然段落,或者加个小标题。
    因为内容不长,可以分两到三个自然段落,或者加一个`

    `作为小节标题(虽然原文没有标题,但规则说“长文用

    /

    划分小节”,如果不长,直接用`

    `包裹也可以,或者为了体现排版,加个隐含的标题?不,规则4要求“不得增删改内容”,所以我不能自己加标题文字。那我就直接用`

    `包裹自然段落。 原文可以划分为几个自然段落: 段落1:Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly access carries no information about the input. 段落2:We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context. At a single transformer layer during prefill, BOS Summary relaxes the causal constraint for the BOS query row alone, letting BOS aggregate all visual and textual input tokens into a self-derived context representation while every other token retains standard causal attention. The enriched representation then remains accessible to the model throughout decoding, with no additional parameters, auxiliary modules, or decoding-time overhead. 段落3:Across multiple VLM and LLM families and eight multimodal and text-only benchmarks, BOS Summary improves over direct and Chain-of-Thought prompting and outperforms recent sink-intervention methods on multi-step reasoning tasks. A mechanistic analysis shows that making BOS input-specific also weakens its sink dominance and redistributes attention toward content-bearing prompt tokens, indicating that changing what a sink carries reshapes how later layers allocate attention. 检查是否有重要句子可以用``。 比如:"We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context." 可以加粗。 或者 "BOS Summary" 可以加粗。为了符合“重要句子可用”,我可以把核心贡献的句子加粗。 段落1:

    Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly access carries no information about the input.

    段落2:

    We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context. At a single transformer layer during prefill, BOS Summary relaxes the causal constraint for the BOS query row alone, letting BOS aggregate all visual and textual input tokens into a self-derived context representation while every other token retains standard causal attention. The enriched representation then remains accessible to the model throughout decoding, with no additional parameters, auxiliary modules, or decoding-time overhead.

    段落3:

    Across multiple VLM and LLM families and eight multimodal and text-only benchmarks, BOS Summary improves over direct and Chain-of-Thought prompting and outperforms recent sink-intervention methods on multi-step reasoning tasks. A mechanistic analysis shows that making BOS input-specific also weakens its sink dominance and redistributes attention toward content-bearing prompt tokens, indicating that changing what a sink carries reshapes how later layers allocate attention.

    检查规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 -> 满足 2. 每个自然段落用

    包裹,长文用

    /

    划分小节。 -> 满足(这里不算特别长,分三个段落用

    即可,不加虚构的标题以免违反“不得增删改内容”) 3. 列表、引用、代码、重要句子。 -> 用了包裹重要句子。 4. 保留原文所有信息,不得增删改内容。 -> 满足,文字完全一致。 5. 不要输出文档级标签。 -> 满足。 再核对一下文字是否有遗漏或修改: 原文: Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly access carries no information about the input. We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context. At a single transformer layer during prefill, BOS Summary relaxes the causal constraint for the BOS query row alone, letting BOS aggregate all visual and textual input tokens into a self-derived context representation while every other token retains standard causal attention. The enriched representation then remains accessible to the model throughout decoding, with no additional parameters, auxiliary modules, or decoding-time overhead. Across multiple VLM and LLM families and eight multimodal and text-only benchmarks, BOS Summary improves over direct and Chain-of-Thought prompting and outperforms recent sink-intervention methods on multi-step reasoning tasks. A mechanistic analysis shows that making BOS input-specific also weakens its sink dominance and redistributes attention toward content-bearing prompt tokens, indicating that changing what a sink carries reshapes how later layers allocate attention. HTML:

    Large vision-language models (LVLMs) and language models (LLMs) often exhibit attention sinks, whereby particular positions, such as the beginning-of-sequence (BOS) token, receive a disproportionate share of attention. Under standard causal masking, however, the BOS query attends only to itself, so the representation that later tokens repeatedly access carries no information about the input.

    We introduce BOS Summary, a training-free inference-time method that repurposes this persistent pathway to carry input context. At a single transformer layer during prefill, BOS Summary relaxes the causal constraint for the BOS query row alone, letting BOS aggregate all visual and textual input token