首页 > AI前沿 > Represented but misread: Post-training changes how language models read out what they represent

Represented but misread: Post-training changes how language models read out what they represent

arXiv自然语言 2026-11-20 02:43 18 阅读 查看原文
用户要求将提供的纯文本新闻正文排版成语义化的HTML片段。 规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 2. 每个自然段落用

包裹,长文用

/

划分小节。 3. 列表内容用
  • ,引用用
    ,代码用
    ,重要句子可用。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出//等文档级标签。
    
    原文:
    Does post-training change what a language model represents, or how this representation is read out into an answer? We compared linear probes with the models' own answers, analyzed with signal detection theory, in six open-weight base models and their post-trained versions. For judgments of whether German texts were written by a human or an AI, probes and word n-gram classifiers separated the classes well, but the answers did not and often ordered the texts inversely. Post-training lowered the sensitivity of the answers on average ($Δd' = -0.34$ across model families; -0.29 across three prompts). It also changed the use of a stated base rate: Base models shifted their log-odds by less than the stated change in prior odds, most post-trained models by more. In two models with stage-wise checkpoints, the change occurred during supervised fine-tuning, model stitching implicated the early layers, and from mid-depth on internal states at positions after the text, not those of the text, transferred the change between checkpoints. Our corpora cannot be shared, so we add a case reproducible from public data, the truth of negated statements. Here, post-training acted in the opposite direction: Probes decoded truth at or near ceiling in every model, the answers of several base models did not use it, and post-training repaired them; by mid-depth the repair resided at the positions after the statement; below it, the representation of the statement contributed, strongly in one of four models. In a training study, adding 2,000 correctly labeled authorship judgments of news texts to otherwise identical fine-tuning data improved the answers on our newspaper and textbook corpora and inverted them on the chats, where they followed a news-trained probe of the base model. Post-training thus changed how a judgment is read out of what a model represents, and a small training signal can set its direction.
    
    分析原文结构:
    原文是一个长段落,但逻辑上可以分为几个小节。不过用户说“长文用

    /

    划分小节”,这里原文其实没有明显的小节标题。如果强行加标题会违反“不得增删改内容”的规则。所以最好保持为一个或几个

    ,或者根据逻辑断句但不加标题(因为加标题属于增内容,除非从原文提取,但原文没有标题)。 等等,规则说“长文用

    /

    划分小节”,如果原文本身没有小节,我不加标题是否违反规则?“长文用...划分小节”意味着如果有小节才用,或者我可以提取主题作为标题?但规则4说“不得增删改内容”。为了安全,不添加原文没有的标题,只把原文的自然段落用

    包裹。原文只有一个自然段落。 等等,原文只有一个自然段落。那我就用一个

    包裹整个段落? 或者把数学公式用包裹?$Δd' = -0.34$ 可以变成 $Δd' = -0.34$。 重要句子可用。比如第一句“Does post-training change what a language model represents, or how this representation is read out into an answer?” 或者最后一句“Post-training thus changed how a judgment is read out of what a model represents, and a small training signal can set its direction.” 可以加。 检查规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释; 2. 每个自然段落用

    包裹,长文用

    /

    划分小节; 3. 列表内容用
    • ,引用用
      ,代码用
      ,重要句子可用;
      4. 保留原文所有信息,不得增删改内容;
      5. 不要输出//等文档级标签。
      
      原文只有一个自然段。我直接用

      包裹。 里面包含公式 $Δd' = -0.34$,可以用 $Δd' = -0.34$ 或者 。用 比较合适,或者直接保留文本,因为不是代码。用 也没问题。 “重要句子可用”,我把首句和尾句加。 构建HTML:

      Does post-training change what a language model represents, or how this representation is read out into an answer? We compared linear probes with the models' own answers, analyzed with signal detection theory, in six open-weight base models and their post-trained versions. For judgments of whether German texts were written by a human or an AI, probes and word n-gram classifiers separated the classes well, but the answers did not and often ordered the texts inversely. Post-training lowered the sensitivity of the answers on average ($Δd' = -0.34$ across model families; -0.29 across three prompts). It also changed the use of a stated base rate: Base models shifted their log-odds by less than the stated change in prior odds, most post-trained models by more. In two models with stage-wise checkpoints, the change occurred during supervised fine-tuning, model stitching implicated the early layers, and from mid-depth on internal states at positions after the text, not those of the text, transferred the change between checkpoints. Our corpora cannot be shared, so we add a case reproducible from public data, the truth of negated statements. Here, post-training acted in the opposite direction: Probes decoded truth at or near ceiling in every model, the answers of several base models did not use it, and post-training repaired them; by mid-depth the repair resided at the positions after the statement; below it, the representation of the statement contributed, strongly in one of four models. In a training study, adding 2,000 correctly labeled authorship judgments of news texts to otherwise identical fine-tuning data improved the answers on our newspaper and textbook corpora and inverted them on the chats, where they followed a news-trained probe of the base model. Post-training thus changed how a judgment is read out of what a model represents, and a small training signal can set its direction.

      检查是否有增删改: 原文: Does post-training change what a language model represents, or how this representation is read out into an answer? We compared linear probes with the models' own answers, analyzed with signal detection theory, in six open-weight base models and their post-trained versions. For judgments of whether German texts were written by a human or an AI, probes and word n-gram classifiers separated the classes well, but the answers did not and often ordered the texts inversely. Post-training lowered the sensitivity of the answers on average ($Δd' = -0.34$ across model families; -0.29 across three prompts). It also changed the use of a stated base rate: Base models shifted their log-odds by less than the stated change in prior odds, most post-trained models by more. In two models with stage-wise checkpoints, the change occurred during supervised fine-tuning, model stitching implicated the early layers, and from mid-depth on internal states at positions after the text, not those of the text, transferred the change between checkpoints. Our corpora cannot be shared, so we add a case reproducible from public data, the truth of negated statements. Here, post-training acted in the opposite direction: Probes decoded truth at or near ceiling in every model, the answers of several base models did not use it, and post-training repaired them; by mid-depth the repair resided at the positions after the statement; below it, the representation of the statement contributed, strongly in one of four models. In a training study, adding 2,000 correctly labeled authorship judgments of news texts to otherwise identical fine-tuning data improved the answers on our newspaper and textbook corpora and inverted them on the chats, where they followed a news-trained probe of the base model. Post-training thus changed how a judgment is read out of what a model represents, and a small training signal can set its direction. 对比:

      Does post-training change what a language model represents, or how this representation is read out into an