首页 > AI前沿 > MoE-Context: Show the Model Only What It Needs

MoE-Context: Show the Model Only What It Needs

arXiv自然语言 2026-11-18 21:08 17 阅读 查看原文
用户要求将一段纯文本新闻正文排版成语义化的HTML片段。 需要遵循以下规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/

AI agents read their whole history at every step. People do not work like that: we keep what the moment needs and hold the rest as a rough note we can recall. MoE-Context applies this to language models. The context is split into blocks. Before each step a fast yes or no pass decides which blocks the step needs; the rest stay as one-line notes that can be brought back, and nothing is deleted. In the version we build, one model makes this decision and then answers, over a single key-value cache where hidden blocks are masked rather than removed.

实验结果与模型表现

On a laptop with Qwen3-1.7B and a synthetic lookup task, showing only the needed block raises accuracy from 93 to 119 of 120, and a masked block is as hidden as a deleted one. The model's own picking is the weak part. A supervised linear probe written into spare output rows lifts its accuracy from 80 to 110 of 120, but simple keyword search picks perfectly on this task, and the probe does not transfer to a second task.

应用与展望

We report these results openly, along with two cheap usage patterns that block switching enables:

  • undoing a failed attempt
  • running subagents without a new context
.

等一下,规则说“保留原文所有信息,不得增删改内容”。如果我自己加小标题,算不算“增改内容”? 规则2说:“长文用

/

划分小节”,如果我加小标题,就增加了原文没有的文字。为了严格遵守“不得增删改内容”,最好不添加原文没有的标题文字,或者只使用纯HTML标签,不加自定义的标题文本。但如果没有标题,怎么用`

`/`

`? 也许可以把第一句或者某些句子作为标题?不,那也改变了结构。 或者整个文本就作为一个或几个`

`,不需要强行加`

`,因为规则说“长文用”,这段并不算特别长。 或者,我可以直接输出几个`

`,不添加任何原文没有的文字。 原文本: 段落1:AI agents read their whole history at every step. People do not work like that: we keep what the moment needs and hold the rest as a rough note we can recall. MoE-Context applies this to language models. The context is split into blocks. Before each step a fast yes or no pass decides which blocks the step needs; the rest stay as one-line notes that can be brought back, and nothing is deleted. In the version we build, one model makes this decision and then answers, over a single key-value cache where hidden blocks are masked rather than removed. 段落2:On a laptop with Qwen3-1.7B and a synthetic lookup task, showing only the needed block raises accuracy from 93 to 119 of 120, and a masked block is as hidden as a deleted one. The model's own picking is the weak part. A supervised linear probe written into spare output rows lifts its accuracy from 80 to 110 of 120, but simple keyword search picks perfectly on this task, and the probe does not transfer to a second task. 段落3:We report these results openly, along with two cheap usage patterns that block switching enables: undoing a failed attempt and running subagents without a new context. 如果我不加自创的标题,只用`

`,是符合“不得增删改内容”的。 如果非要用`

`,可以把某些句子提取出来当标题?不,那属于增改。 所以最安全的做法是直接使用`

`分段,对重要数据使用``。 检查文本: "AI agents read their whole history at every step. People do not work like that: we keep what the moment needs and hold the rest as a rough note we can recall. MoE-Context applies this to language models. The context is split into blocks. Before each step a fast yes or no pass decides which blocks the step needs; the rest stay as one-line notes that can be brought back, and nothing is deleted. In the version we build, one model makes this decision and then answers, over a single key-value cache where hidden blocks are masked rather than removed." -> 第一个`

` "On a laptop with Qwen3-1.7B and a synthetic lookup task, showing only the needed block raises accuracy from 93 to 119 of 120, and a masked block is as hidden as a deleted one. The model's own picking is the weak part. A supervised linear probe written into spare output rows lifts its accuracy from 80 to 110 of 120, but simple keyword search picks perfectly on this task, and the probe does not transfer to a second task." -> 第二个`

` "We report these results openly, along with two cheap usage patterns that block switching enables:

  • undoing a failed attempt
  • running subagents without a new context
." -> 第三个`

`,包含列表。注意原文是 "undoing a failed attempt and running subagents without a new context." 改成列表是否算“改内容”? 规则3说“列表内容用

  • ”,原文中这部分是并列的,用列表呈现是排版需要,不算改变文字内容,但标点符号可能需要调整。原文是 "undoing a failed attempt and running subagents without a new context." 如果改成列表:
    • undoing a failed attempt
    • running subagents without a new context
    这样去掉了 "and" 和句号。为了严格保留文字,可以这样:
    • undoing a failed attempt
    • running subagents without a new context.
    或者保留and: ...enables:
    • undoing a failed attempt
    • and running subagents without a new context
    还是保持自然一点: ...enables:
    • undoing a failed attempt
    • running subagents without a new context
    其实把 "and" 放在列表项里或者去掉都算排版调整。为了最严谨,我直接把整个句子放在`

    `里,列表项不带句号,或者在最后一项带句号。 "We report these results openly, along with two cheap usage patterns that block switching