首页 > AI前沿 > Deep Latents, Long Memory: Latent persistence explains temporal correlations in transformer activations

Deep Latents, Long Memory: Latent persistence explains temporal correlations in transformer activations

arXiv自然语言 2026-11-13 20:14 16 阅读 查看原文
用户现在需要把给定的新闻正文(其实是学术文本)排版成语义化的HTML片段,规则要遵守,只输出HTML,不用文档级标签,每个自然段用p,长文用h2/h3,这里有没有适合的小节?看原文是一段?不对,原文是一段?不对,看原文内容,哦,原文是一个整体?不对,仔细看,原文是一段连贯的内容,但可以划分吗?等下,原文内容是关于大语言模型隐藏层的自相关函数的研究,首先,有没有适合的h2?比如可能可以分?不对,再看规则,长文用h2/h3划分小节,原文这里是一个整体?不对,看原文的段落,原文是一段吗?对,给定的正文是一段连贯的内容?不对,再仔细看用户给的正文,是一段?哦,不对,用户给的正文是一整段?那怎么办?不对,等下,有没有可以划分的?比如,首先,开头提出问题,然后提出方法,然后合成数据结果,然后真实文本结果,然后补充分析?可以划分h2小节吗?比如h2是研究背景与方法?然后h2是合成数据实验结果?h2是真实文本实验结果?h2是补充验证?不对,但是要保证不增删改内容,不能改原文内容。或者,先看原文的结构:第一句到“separated by a lag.”是背景和定义,然后接下来是合成数据的结果,然后是真实文本的结果,然后是补充分析。那可以加h2划分吗?对,按照语义划分小节,同时保留所有内容。等下,还要注意,不能增删内容,所以h2的小节标题是概括性的,不能加原文没有的?不对,不对,规则2是长文用h2/h3划分小节,但是小节标题需要是原文有的吗?哦,不对,原文没有小节标题,那怎么办?哦,等下,再仔细看用户给的正文,是不是就是一段?不对,用户给的正文是: “The hidden layers of Large Language Models (LLMs) are believed to encode representations of the data whose abstraction increases with depth. However, in the absence of an operational notion of abstraction, this statement is hard to test quantitatively. Here, we propose an alternative view, building on the principle that the correlations of a sequence reflect the structure of the process that generated it. We define the autocorrelation function of a layer as the average dot product between its activations at two context positions separated by a lag. Latent variables encoded persistently along the context create slowly decaying tails in the autocorrelation, and we find these tails to grow across layers in both synthetic and natural data. In synthetic data with controlled hierarchical latent structure, this growth stems from the encoding of abstract latent variables, which govern long stretches of the input. Indeed, the persistence of latent codes predicts the qualitative structure of autocorrelations and accounts for most of the empirical measures. In real text, we take sparse-autoencoder features as proxies for latent variables, and find that their persistence correlates strongly with the magnitude of autocorrelations across datasets and LLMs. In particular, the persistence time increases with depth, as do autocorrelations, implying that deeper latent features persist for longer context spans. We complement this analysis by explicitly ablating the persistent sparse-autoencoder features and observing a consequent decrease in the autocorrelations.” 哦,原来这是一个连续的段落,但是可以按照语义划分h2吗?比如,第一个h2是“研究背景与方法”,里面放前面的句子,然后第二个h2是“合成数据实验结果”,第三个h2是“真实文本实验结果”,第四个h2是“补充验证分析”?不对,但是规则是不能增删改内容,那h2的标题是新加的?会不会不符合?哦,不对,再看规则2:“每个自然段落用

包裹,长文用

/

划分小节”,这里的长文划分小节,是指对内容的语义划分,但是小节标题应该是原文的逻辑划分的概括?还是说,如果原文没有小节标题,就不用?不对,不对,再仔细看,有没有可能原文就是一段,那直接用p包裹?不对,等下,再看内容里有没有可以加强调的,比如重要的句子?比如,核心的定义?或者研究结论?不对,规则里说重要句子可用strong。那首先,先处理: 首先,有没有h2?比如,这段内容可以分为几个部分: 1. 研究背景与方法定义:从开头到“separated by a lag.”——可以用h2包装,标题比如“研究背景与方法”?不对,但是会不会算增内容?哦,不对,等下,用户说“保留原文所有信息,不得增删改内容”,那加的h2标题是不是增内容?哦,对哦,规则4是不能增删改,那不能加原文没有的文字?那怎么办?哦,哦,不对,我理解错了?等下,规则2的“用

/

划分小节”,是不是指如果原文有小节标题的话用,如果没有的话,就不用?不对,原文是纯文本的新闻正文?哦,用户开头说“纯文本新闻正文”,哦,用户说的是新闻正文,那可能我误以为是学术论文,但其实是新闻?哦,不对,内容是学术论文的内容,但是用户说是新闻正文。那再看,原文有没有自然段落?用户给的正文是一整段?那如果是一整段,就用一个p?不对,但是里面有没有列表?没有,有没有引用?没有,有没有代码?没有。那重要句子可以加strong?比如,核心的定义?或者核心结论? 等下,再仔细想规则:不能增删改内容,所以不能加任何原文没有的文字,包括h2的标题,因为原文没有这些标题,所以不能加?那这样的话,原文是一个完整的自然段落,所以用一个

包裹?但是不对,长文用h2/h3划分,那如果原文没有小节,就不需要?不对,再看内容,有没有可以划分的?哦,不对,原文的内容是连贯的,没有分小节,那是不是直接用p包裹?但是还要加strong,比如重要的句子,比如“We define the autocorrelation function of a layer as the average dot product between its activations at two context positions separated by a lag.”这个是核心定义,可以加strong?还有“we find these tails to grow across layers in both synthetic and natural data.”是核心发现?还有“the persistence time increases with depth, as do autocorrelations, implying that deeper latent features persist for longer context spans.”也是核心结论? 不对,等下,再确认规则:4.保留原文所有信息,不得增删改内容。所以不能加任何原文没有的内容,包括h2的标题,因为原文没有,所以不能加h2?那如果原文是一段,就直接

包裹全部内容,里面适当加strong? 不对,等下,再看用户给的正文,是不是真的是一段?是的,用户给的正文是连续的一段,没有分段。那现在处理: 首先,整个内容用

包裹?然后里面的重要句子加strong。比如:

The hidden layers of Large Language Models (LLMs) are believed to encode representations of the data whose abstraction increases with depth. However, in the absence of an operational notion of abstraction, this statement is hard to test quantitatively. Here, we propose an alternative view, building on the principle that the correlations of a sequence reflect the structure of the process that generated it. We define the autocorrelation function of a layer as the average dot product between its activations at two context positions separated by a lag. Latent variables encoded persistently along the context create slowly decaying tails in the autocorrelation, and we find these tails to grow across layers in both synthetic and natural data. In synthetic data with controlled hierarchical latent structure, this growth stems from the encoding of abstract latent variables, which govern long stretches of the input. Indeed, the persistence of latent codes predicts the qualitative structure of autocorrelations and accounts for most of the empirical measures. In real text, we take sparse-autoencoder features as proxies for latent variables, and find that their persistence correlates strongly with the magnitude of autocorrelations across datasets and LLMs. In particular, the persistence time increases with depth, as do autocorrelations, implying that deeper latent features persist for longer context spans. We complement this analysis by explicitly ablating the persistent sparse-autoencoder features and observing a consequent decrease in the autocorrelations.

不对,但是规则2说长文用h2/h3划分小节,那如果原文是长文,但没有小节标题,怎么处理?哦,会不会我理解错了,h2/h3是划分内容的层级,不需要加额外的文字,而是把内容分到不同的h2下面?不对,h2需要有内容啊,h2里面是标题,然后是p。如果加标题就是增内容了,违反规则4。哦,对哦,规则4明确说不得增删改内容,所以不能加任何原文没有的文字,所以