首页 > AI前沿 > The Thinking Leakage in Reinforcement Learning for LLMs

The Thinking Leakage in Reinforcement Learning for LLMs

arXiv自然语言 2026-10-10 03:43 4 阅读 查看原文

Hybrid reasoning models such as the Qwen3 series switch between a THINK mode and a NO-THINK mode through a few chat-template tokens.

A popular recipe trains such models with reinforcement learning in NO-THINK mode and reports improved reasoning performance.

We provide evidence that much of this gain can be explained by thinking leakage: the gain comes largely from partially reactivating the reasoning abilities of the original model's THINK mode instead of learning enhanced reasoning behavior.

We train Qwen3-1.7B, 4B, and 8B with GRPO in NO-THINK mode.

As accuracy improves, responses become longer and more reflective, are increasingly judged by the original model to resemble its own thinking, and the model's internal activations shift toward its THINK mode.

We then investigate at the level of weights, activations, and attention heads:

We further identify a possible mechanism in the GRPO training signal: responses favored by the advantage-weighted update are more THINK-like, and a gradient step computed from this signal moves the model toward its THINK state.

Thus, gains in NO-THINK mode can largely reflect reactivation of reasoning abilities already present in the original model.