首页 > AI前沿 > The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices

The Truncation Blind Spot: How Decoding Strategies Systematically Exclude Human-Like Token Choices

arXiv机器学习 2026-03-19 12:36 6 阅读 查看原文

Why does machine-generated text remain detectable? We investigate a mechanistic explanation at the decoding stage:

standard strategies such as top-$k$ and nucleus sampling restrict generation to high-probability tokens, while human writers routinely choose contextually appropriate words from deeper in the model's probability distribution.

Truncation makes a measurable share of these choices unreachable; we call this the truncation blind spot.

Across five open models and three domains, 8--18% of human-selected tokens fall outside common truncation boundaries.

Linguistic analysis further reveals disproportionate exclusion of content-word tokens.

In a benchmark comprising 1.8 million machine generations, classifiers using only predictability and lexical diversity achieve mean AUC-ROC near 0.97, with substantial variation across decoding settings and strong transfer across generators.

Probability-floor samplers substantially narrow the blind spot, demonstrating that the choice of truncation criterion matters for retaining human-used tokens.

Together, these findings characterize a source of human--machine distributional mismatch and motivate decoding methods that preserve contextually appropriate low-probability choices while maintaining generation quality.

Code and data are available at https://github.com/EstebanGarces/human_vs_machine.