Why do language models fail on long inputs?
The common answer is length. We show that, for retrieval, length is cheap and competition is not. On five open models (0.5B to 3B parameters, four families), neutral filler costs nothing: accuracy stays at 0.97 to 1.00, up to 32K tokens on Qwen2.5-1.5B. Statements in the format of the evidence, placed at the same positions, drop it to 0.16 to 0.33.
- A single statement that gives the queried key a rival value multiplies the loss by 9 to 110.
- Every additional rival adds to the loss.
Model Internals Explaining Failure
Inside the model, one quantity tracks the failure: the attention mass left on the evidence. Across 126,473 trials, it explains up to 58% of the variance in loss; log length explains at most 8%.
Masked-Filler Protocol Results
The masked-filler protocol behind the claim that length alone hurts reproduces on MMLU (11 to 18 points lost at 30K tokens), but fails a visible control in retrieval:
- Masking costs 45 accuracy points at 32K tokens.
- The same filler left visible costs none.
Conclusion: Context Rot as Attention Competition
In retrieval, context rot is a competition for attention. We release the probe and all 126K trial-level measurements.