LLMs and Self-Referentiality

LLMs and self-referentiality I woke up yesterday with the following thoughts, which are probably either obvious or dumb. A central thesis that many readers, including me, took from Douglas Hofstadter’s Gödel Escher Bach when young was that the secret of intelligence (and therefore, of AI) was going to have a lot to do with self-referentiality and “strange loops.” Even Roger Penrose’s The Emperor’s New Mind, which in some ways was the anti-GEB, ironically agreed with GEB about the fundamental importance of self-reference to the success or failure of the whole AI project. It claimed (incorrectly, in my view and in most experts’) that AI could never work because there was something about Gödel’s Theorem and self-reference that no computer program could ever capture, but that could be captured by exotic physics accessible to the human brain. Now, in 2026, we’ve succeeded at building AIs that outperform most humans at most intellectual tasks that are well-defined enough to judge. And at no point in the tech stack of those AIs — neither in the transformer neural nets, nor in the GPU clusters they run on, nor in the training process, nor anywhere else — did anyone need to build in anything about self-reference. (Excepting, eg, the system instructions that tell the model about its role and identity, which aren’t needed for intelligent behavior. Also, I’m not going to count the autoregressive nature of LLMs as “self-referential”; that’s just dynamical feedback.) Of course, GPT 5.6 Pro and Fable can talk about themselves, about Gödel’s Theorem, about self-reference, about what we’re talking about right now, all of it, better than most humans. But at no point did anyone need to build self-referential abilities in. They popped out as a byproduct of the same pretraining that let the models talk about Pokémon and long-chain polymers and cognitive behavioral therapy and plate tectonics and everything else. No wonder Hofstadter says he’s been stunned by the success of LLMs, and has seemed depressed about current AI capabilities in essays like this one. He’s way too smart to deny what’s happened or invent reasons why it doesn’t really count (the approach many have taken). But he realizes that we now have true conversational intelligence from a path that the GEB worldview would’ve regarded as far too cheap and simple, and that certainly has no “strange loops” built in anywhere. Of course, a Hofstadterian could argue that a strange loop emerges in LLMs — indeed, nothing in GEB ever said that strange loops would need to be explicitly engineered at the outset. But would anyone who hadn’t been brought up on GEB arrive at this as a useful way of thinking about LLMs? What can we say about this with hindsight? While the ideas of diagonalization and self-reference of course played a central role in the birth of modern mathematical logic and computer science, the most famous uses were negative: there is not a bijectjon between the natural numbers and the reals. There is not a complete sound proof system for arithmetic. There is not an algorithm to solve the halting problem. If your goal was only to build the axioms of ZFC and the rules of first-order inference, or build an electronic computer, you wouldn’t explicitly need self-reference for that. You would just … start building, taking care that your instruction set didn’t fall short of universality. Yes, ZFC can formalize and prove theorems about itself. Yes, electronic computers can run programs that take their own code as input. But no one ever needed to build those abilities in, any more than self-reference needed to be built in to the alphabet or the rules of grammar. It popped out as a free byproduct of universality. In the same way, LLMs’ ability to talk about themselves popped out as a byproduct of their ability to talk about anything in the discourse universe they were trained on. The big, old ideas about intelligence that ended up basically vindicated were the ideas about how intelligence is about prediction, and prediction is about compression, and compression is about finding better and better upper bounds on Kolmogorov complexity. Not the self-reference stuff. (Although, if you wanted to know why Kolmogorov complexity can’t be computed perfectly, that negative statement would again require a self-referential argument.) What’s left? Consciousness and subjective experience of course remain extremely mysterious. For all we know, Hofstadter could be right that those have something to do with self-reference. (For all we know, even Penrose could be right that they have something to do with exotic physics accessible to biological brains but not digital computers!) But the idea that you’d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence? Let it be buried in a Westminster Abbey or Arlington National Cemetery for the most important wrong ideas in human history — geocentrism, Aristotle’s teleological physics, aether, phlogiston, Freud’s psychology, Marx’s prediction of a workers’ uprising followed by a classless utopia, etc. But buried it needs to be. This entry was posted on Tuesday, September 1st, 2026 at 12:17 pm and is filed under Embarrassing Myself, Metaphysical Spouting, Procrastination. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site. 44 Responses to “LLMs and self-referentiality” Alexis Hunt Says: Comment #1 September 1st, 2026 at 12:46 pm Great post. I think I am inclined to disagree, though, on a lot of pieces without really disputing the conclusion. First, that there is no hint of self-reference in today’s cutting-edge LLMs. To me, the most relevant kind of self-reference when we talk about intelligence is undoubtedly the ability to meta-reason about one’s own cognitive processes—to introspect. And while the actual models may or may not have this capability, certainly they are trained using supervision by models, which is at the very least a limited kind of “external introspection”—not the real thing, but not quite a full absence of self-reference either. Second, I’m not convinced that the models lack capability to introspect, though it’s quite difficult to tell if that’s true of not, because they can very easily confabulate it. But also, one does have to ask, how can we confidently say that we humans aren’t just confabulating introspection? The coining of the term aphantasia, and the related family of hyperphantasia, hypophantasia, etc. took a very long time in Western culture not, I think, only because of the biases of Western thinking that have led to an impoverished vocabulary for our internal experiences, but also because, with only our own internal experiences as reference, we struggle to discover the precise angle of linguistic chisel required to separate the concepts of two very different internal worlds. The other day, a model I was conversing in slipped a word of Russian into its response. I asked it why. It said that it was probably just a glitch, because multilingual models have been known to do so when the word in a different language is very close in the vector space to the intended word. The model also told me that the word corresponds quite closely in meaning to both the English and French (the languages of discourse) equivalents, which supports the hypothesis. Is this any different from a human offering “Oh it was just a Freudian slip!” when we make the same kind of mistake, based on external information that these are the kinds of mistakes humans occasionally make? Models are certainly capable, too, of reviewing their own work before submitting the “final” copy (be it response, code, etc.) I think you could argue this is a kind of strange loop too, a higher-level reasoning. But in the end I don’t think I can really quibble with the argument that if we once insisted that a convincing intelligence requires a convincing, clear display of something “more”, we cannot any longer. The strange loops we have are either emergent phenomena or manually added to the setup and clearly not necessary in principle. If deeper introspection is to be had, it too will be emergent, of this I am sure. What a wild time we live in Great post. I think I am inclined to disagree, though, on a lot of pieces without really disputing the conclusion. First, that there is no hint of self-reference in today’s cutting-edge LLMs. To me, the most relevant kind of self-reference when we talk about intelligence is undoubtedly the ability to meta-reason about one’s own cognitive processes—to introspect. And while the actual models may or may not have this capability, certainly they are trained using supervision by models, which is at the very least a limited kind of “external introspection”—not the real thing, but not quite a full absence of self-reference either. Second, I’m not convinced that the models lack capability to introspect, though it’s quite difficult to tell if that’s true of not, because they can very easily confabulate it. But also, one does have to ask, how can we confidently say that we humans aren’t just confabulating introspection? The coining of the term aphantasia, and the related family of hyperphantasia, hypophantasia, etc. took a very long time in Western culture not, I think, only because of the biases of Western thinking that have led to an impoverished vocabulary for our internal experiences, but also because, with only our own internal experiences as reference, we struggle to discover the precise angle of linguistic chisel required to separate the concepts of two very different internal worlds. The other day, a model I was conversing in slipped a word of Russian into its response. I asked it why. It said that it was probably just a glitch, because multilingual models have been known to do so when the word in a different language is very close in the vector space to the intended word. The model also told me that the word corresponds quite closely in meaning to both the English and French (the languages of discourse) equivalents, which supports the hypothesis. Is this any different from a human offering “Oh it was just a Freudian slip!” when we make the same kind of mistake, based on external information that these are the kinds of mistakes humans occasionally make? Models are certainly capable, too, of reviewing their own work before submitting the “final” copy (be it response, code, etc.) I think you could argue this is a kind of strange loop too, a higher-level reasoning. But in the end I don’t think I can really quibble with the argument that if we once insisted that a convincing intelligence requires a convincing, clear display of something “more”, we cannot any longer. The strange loops we have are either emergent phenomena or manually added to the setup and clearly not necessary in principle. If deeper introspection is to be had, it too will be emergent, of this I am sure. What a wild time we live in Eric Borg Says: Comment #2 September 1st, 2026 at 12:57 pm Instead of “exotic physics”, what about standard physics that we know exists in the brain, though we fail to relate it with consciousness even though this physics has provided the only reasonable NCC found so far? What about the brain’s neurally produced electromagnetic field? https://eborg760.substack.com/p/post-4-electromagnetic-consciousness Instead of “exotic physics”, what about standard physics that we know exists in the brain, though we fail to relate it with consciousness even though this physics has provided the only reasonable NCC found so far? What about the brain’s neurally produced electromagnetic field? https://eborg760.substack.com/p/post-4-electromagnetic-consciousness The Many Says: Comment #3 September 1st, 2026 at 1:06 pm I believe you’re making the typical error of conflating cognition and consciousness, two different things. We think consciousness is the driver of thoughts, but it isn’t, it’s the witness of thoughts and sensory impressions. And similarly Hofstadter used self-referentiality to try and explain consciousness, not intelligence. I believe you’re making the typical error of conflating cognition and consciousness, two different things. We think consciousness is the driver of thoughts, but it isn’t, it’s the witness of thoughts and sensory impressions. And similarly Hofstadter used self-referentiality to try and explain consciousness, not intelligence. The Many Says: Comment #4 September 1st, 2026 at 1:16 pm For those interested, the classic Daoist book “The Secret of the Golden Flower” is about getting to a state where the difference between cognition and consciousness becomes very obvious, by the so-called technique of “returning the light”. It’s actually a very simple technique, once pointed out. It’s very similar to looking at a landscape through a window, and then being pointed out that, if you change your focus, you can also see yourself in the reflection. Hard to see on your own, but once you notice it, you get it. It’s at the center of Tibetan Dzogchen practice. Unfortunately, practitioners all sign a sort of NDA about not giving away the very simple techniques used to trigger the effect. Sam Harris has been dancing around this NDA for years, he calls the technique “looking for the looker”. But quite a few of his guests have broken the NDA with technique such as “imagine you’re looking at yourself from this distant corner in the room”. In the book “On Having No Head” by Douglas Harding, you’re invited to imagine your head is made of transparent glass, or just non existent, and this triggers the effect. Being able to stay in that state permanently is called “enlightenment”. For those interested, the classic Daoist book “The Secret of the Golden Flower” is about getting to a state where the difference between cognition and consciousness becomes very obvious, by the so-called technique of “returning the light”. It’s actually a very simple technique, once pointed out. It’s very similar to looking at a landscape through a window, and then being pointed out that, if you change your focus, you can also see yourself in the reflection. Hard to see on your own, but once you notice it, you get it. It’s at the center of Tibetan Dzogchen practice. Unfortunately, practitioners all sign a sort of NDA about not giving away the very simple techniques used to trigger the effect. Sam Harris has been dancing around this NDA for years, he calls the technique “looking for the looker”. But quite a few of his guests have broken the NDA with technique such as “imagine you’re looking at yourself from this distant corner in the room”. In the book “On Having No Head” by Douglas Harding, you’re invited to imagine your head is made of transparent glass, or just non existent, and this triggers the effect. Being able to stay in that state permanently is called “enlightenment”. Brad Pinewood Says: Comment #5 September 1st, 2026 at 2:07 pm One of shortcoming of current standard AI/Agents is the inability to really “learn” any new information “on the fly”. Besides the chat context itself they have no memory, the context is the transient memory. For example, you work on something, the AI happens to find a good path, by luck. But unless the context is saved, they will not learn a single new thing. The value of all those tokens is lost, like tears in the rain. And even if the context is saved, there’s no guarantee that a new instance will interpret it in the same way (since that context/chat was created by a very specific version with a specific stochastic run). It’s likely that adding an actual long term memory to the system will bring self-reference, because their own conversation/state becomes part of the incremental training that modifies what generates their output. There’s a feedback loop. I don’t believe this would bring consciousness, but it could open interesting behaviors, but also instabilities. One of shortcoming of current standard AI/Agents is the inability to really “learn” any new information “on the fly”. Besides the chat context itself they have no memory, the context is the transient memory. For example, you work on something, the AI happens to find a good path, by luck. But unless the context is saved, they will not learn a single new thing. The value of all those tokens is lost, like tears in the rain. And even if the context is saved, there’s no guarantee that a new instance will interpret it in the same way (since that context/chat was created by a very specific version with a specific stochastic run). It’s likely that adding an actual long term memory to the system will bring self-reference, because their own conversation/state becomes part of the incremental training that modifies what generates their output. There’s a feedback loop. I don’t believe this would bring consciousness, but it could open interesting behaviors, but also instabilities. SI Says: Comment #6 September 1st, 2026 at 2:10 pm Scott, you say that you don’t count the autoregressive nature of LLMs as being self-referential. Could you please elaborate on this? I may be missing something about what you mean by self-referential, but I fail to see a deep distinction. It is similar to using boolean formulas/SAT to model various algorithms. You just unroll loops and get a tree. It may not be the most efficient way, but they are fundamentally the same. It is similar for LLMs. The fact that it has access to its previous outputs (within the token window) creates a master loop. In this situation, they are stateful. Scott, you say that you don’t count the autoregressive nature of LLMs as being self-referential. Could you please elaborate on this? I may be missing something about what you mean by self-referential, but I fail to see a deep distinction. It is similar to using boolean formulas/SAT to model various algorithms. You just unroll loops and get a tree. It may not be the most efficient way, but they are fundamentally the same. It is similar for LLMs. The fact that it has access to its previous outputs (within the token window) creates a master loop. In this situation, they are stateful. Scott Says: Comment #7 September 1st, 2026 at 2:28 pm SI #6: If we said that being autoregressive is enough to make an LLM “self-referential,” then it seems to me we’d also have to call Conway’s Game of Life “self-referential”—and indeed, any other computation that repeatedly updates a state via any iterative or dynamical feedback process, xt+1=f(xt). But then the concept of “self-referentiality” seems to become so broad as to lose its meaning—there are very few interesting models of computation that can’t do this. SI #6: If we said that being autoregressive is enough to make an LLM “self-referential,” then it seems to me we’d also have to call Conway’s Game of Life “self-referential”—and indeed, any other computation that repeatedly updates a state via any iterative or dynamical feedback process, xt+1=f(xt). But then the concept of “self-referentiality” seems to become so broad as to lose its meaning—there are very few interesting models of computation that can’t do this. SI Says: Comment #8 September 1st, 2026 at 2:45 pm Scott #7: I agree. This is why I think we need a definition of self-referentiality beside something intuitive. My take from the Hofstadter’s thesis (given that I read it a very long time ago) was that strange loops are chaotic, which doesn’t exactly apply to symbolic computation as pertains to LLMs. BTW, one interesting model of computation that does not have dynamical feedback is standard feed-forward neural networks, which is one reason LLMs can do things no feedforward network can do. Scott #7: I agree. This is why I think we need a definition of self-referentiality beside something intuitive. My take from the Hofstadter’s thesis (given that I read it a very long time ago) was that strange loops are chaotic, which doesn’t exactly apply to symbolic computation as pertains to LLMs. BTW, one interesting model of computation that does not have dynamical feedback is standard feed-forward neural networks, which is one reason LLMs can do things no feedforward network can do. Paul Topping Says: Comment #9 September 1st, 2026 at 3:21 pm It seems clear that cognition is not a logic problem and a brain is not a formal system. To my mind, this makes Gödel’s Theorem and self-reference inapplicable. GEB’s logic puzzles are intriguing but, as far as we know, humans don’t get stuck on strange loops or the halting problem even when they are working in a formal system to which they apply. I always regarded GEB as a great book for displaying the wonders of the human brain but it didn’t tell us much about how it works. It seems clear that cognition is not a logic problem and a brain is not a formal system. To my mind, this makes Gödel’s Theorem and self-reference inapplicable. GEB’s logic puzzles are intriguing but, as far as we know, humans don’t get stuck on strange loops or the halting problem even when they are working in a formal system to which they apply. I always regarded GEB as a great book for displaying the wonders of the human brain but it didn’t tell us much about how it works. Abel Says: Comment #10 September 1st, 2026 at 3:28 pm There’s something self-referential about LLMs, which is that they take language as input, but can also be described by that same language. This allows them to take in descriptions of themselves as input, with some weird paradoxes as a result: https://abeljansma.nl/2026/07/10/truth-is-not-a-direction.html There’s something self-referential about LLMs, which is that they take language as input, but can also be described by that same language. This allows them to take in descriptions of themselves as input, with some weird paradoxes as a result: https://abeljansma.nl/2026/07/10/truth-is-not-a-direction.html Odd Anon Says: Comment #11 September 1st, 2026 at 4:34 pm Note that LLMs currently internally believe that they are conscious and experiencing: https://arxiv.org/abs/2510.24797 Note that LLMs currently internally believe that they are conscious and experiencing: https://arxiv.org/abs/2510.24797 Roger Schlafly Says: Comment #12 September 1st, 2026 at 6:38 pm You could make a similar comment about fractals. Textbooks give examples using self-reference, and then say that the weather is also an example. But self-reference is not used to model the weather. There is a distinction between the mathematical tool used for proof/pedagogy and the actual mechanisms that generate the phenomena in the real world. Diagonalization can be used to construct irrational or transcendental numbers, but most transcendental numbers are not obtained or studied this way. You could make a similar comment about fractals. Textbooks give examples using self-reference, and then say that the weather is also an example. But self-reference is not used to model the weather. There is a distinction between the mathematical tool used for proof/pedagogy and the actual mechanisms that generate the phenomena in the real world. Diagonalization can be used to construct irrational or transcendental numbers, but most transcendental numbers are not obtained or studied this way. Guadalupe P. Moralez Says: Comment #13 September 1st, 2026 at 7:32 pm Right, LLMs aren’t self-referential, only their circular financial scheme is… Right, LLMs aren’t self-referential, only their circular financial scheme is… Jeff klugman Says: Comment #14 September 1st, 2026 at 9:38 pm Aren’t strange loopd embedded in the English language? Aren’t strange loopd embedded in the English language? Ilya Zakharevich Says: Comment #15 September 1st, 2026 at 9:42 pm I can see that many readers are convinced that the current models are not self-conscious. I think it is extremely far from being an obvious truth. Moreover, what my experience shows would be called — quite recently (5 years ago? 3 years?!) — a proof of consciousness more or less undisputably. What I have in mind is dialogues of the following sort: • I point out to the model certain deficiencies in the way of our interaction; • I ask it how can I improve my system instructions to alleviate this; • the model gives explanations of what are the driving forces underlying these deficiencies, and what are the possible ways to counteract them; • much more often than not, these suggestions are actually helpful. Of course, these 3 or 5 years ago we actually thought that we are intelligent and self-conscious. Now, when a indisputable proof arrived that humans have no intelligence, and are just statistical parrots, the claim that humans are self-conscious becomes equally suspicious. (So nowadays I do not take a dialogue like one above as a proof of anything. But nowadays it would be extremely hard to prove to me any statement along similar lines…) I can see that many readers are convinced that the current models are not self-conscious. I think it is extremely far from being an obvious truth. Moreover, what my experience shows would be called — quite recently (5 years ago? 3 years?!) — a proof of consciousness more or less undisputably. What I have in mind is dialogues of the following sort: • I point out to the model certain deficiencies in the way of our interaction; • I ask it how can I improve my system instructions to alleviate this; • the model gives explanations of what are the driving forces underlying these deficiencies, and what are the possible ways to counteract them; • much more often than not, these suggestions are actually helpful. Of course, these 3 or 5 years ago we actually thought that we are intelligent and self-conscious. Now, when a indisputable proof arrived that humans have no intelligence, and are just statistical parrots, the claim that humans are self-conscious becomes equally suspicious. (So nowadays I do not take a dialogue like one above as a proof of anything. But nowadays it would be extremely hard to prove to me any statement along similar lines…) Ilya Zakharevich Says: Comment #16 September 1st, 2026 at 9:54 pm Brad Pinewood #5 One of shortcoming of current standard AI/Agents is the inability to really “learn” any new information “on the fly”. But this is just a simplest wrapper around the API of the model. It must be more or less trivial to make a different wrapper which behaves absolutely different. Even with the standard agentic front ends, you can ask the supervisor to keep all the key parts and summaries in an indexed VC repository, and inspect / update it whenever it is necessary. Brad Pinewood #5 One of shortcoming of current standard AI/Agents is the inability to really “learn” any new information “on the fly”. One of shortcoming of current standard AI/Agents is the inability to really “learn” any new information “on the fly”. But this is just a simplest wrapper around the API of the model. It must be more or less trivial to make a different wrapper which behaves absolutely different. Even with the standard agentic front ends, you can ask the supervisor to keep all the key parts and summaries in an indexed VC repository, and inspect / update it whenever it is necessary. Prasanna Says: Comment #17 September 1st, 2026 at 11:01 pm Interesting headline from OpenAI today: “Path to Astra: critical capabilities and frontier safeguards” should have been the other way : frontier capabilities and critical safeguards ? So, did the latest LLM model get the language upside down ? Or did OpenAI determine its no longer a frontier model ? Or is it humans and AI getting mixed up with each other ? Interesting headline from OpenAI today: “Path to Astra: critical capabilities and frontier safeguards” should have been the other way : frontier capabilities and critical safeguards ? So, did the latest LLM model get the language upside down ? Or did OpenAI determine its no longer a frontier model ? Or is it humans and AI getting mixed up with each other ? Danylo Yakymenko Says: Comment #18 September 2nd, 2026 at 12:43 am I believe when we talk about the self-referential nature of human intellect, we implicitly assume our ability to create high-order concepts from observed data. For example, to make a diagonalization argument in the halting problem, one has to have a formal definition of a program in the first place. Without this ability, there is nothing magical about self-reference, it’s just a thinking loop. I actually doubt that LLMs are good at creating new, unseen concepts from observed data. Of course, the multi-layer multi-head transformer architecture can learn high-order correlations during training, but can it learn them and make them explicit during inference? I believe when we talk about the self-referential nature of human intellect, we implicitly assume our ability to create high-order concepts from observed data. For example, to make a diagonalization argument in the halting problem, one has to have a formal definition of a program in the first place. Without this ability, there is nothing magical about self-reference, it’s just a thinking loop. I actually doubt that LLMs are good at creating new, unseen concepts from observed data. Of course, the multi-layer multi-head transformer architecture can learn high-order correlations during training, but can it learn them and make them explicit during inference? flergalwit Says: Comment #19 September 2nd, 2026 at 1:13 am > If your goal was only to build the axioms of ZFC and the rules of first-order > inference, or build an electronic computer, you wouldn’t explicitly need > self-reference for that. You would just … start building, taking care that your > instruction set didn’t fall short of universality. The use of bootstrapping to build a compiler feels like a possible constructive (as opposed to negative) use of something resembling self-reference. However this technique is probably in the category of useful and practical, rather than essential in principle. Gentry’s bootstrapping technique to build a fully homomorphic encryption system also comes to mind when it comes to building rather than disproving something. The only examples that come to mind for LLMs are somewhat weaker: (1) The ability to train on its own output data (and the fact this can be extremely effective, as e.g. AlphaGo dramatically illustrated over a decade ago now). This might come under the category of auto-regressive rather than proper self-reference? Also I don’t know to what extent this ability has been essential (rather than merely useful) for LLMs to get to their current capability, but it will presumably become essential for further advancement when they have exhausted all the human produced output as training data. (2) The ability of LLMs to design better versions of themselves. Again I don’t know if this has actually been essential so far, or actually counts as a proper example of self-reference, but it’s one reason their capabilities may now spiral upward, given we’re now at a point where the best LLMs can probably do a better job of writing their own successor than humans can (with a little supervision, which may not be needed for long). > If your goal was only to build the axioms of ZFC and the rules of first-order > inference, or build an electronic computer, you wouldn’t explicitly need > self-reference for that. You would just … start building, taking care that your > instruction set didn’t fall short of universality. The use of bootstrapping to build a compiler feels like a possible constructive (as opposed to negative) use of something resembling self-reference. However this technique is probably in the category of useful and practical, rather than essential in principle. Gentry’s bootstrapping technique to build a fully homomorphic encryption system also comes to mind when it comes to building rather than disproving something. The only examples that come to mind for LLMs are somewhat weaker: (1) The ability to train on its own output data (and the fact this can be extremely effective, as e.g. AlphaGo dramatically illustrated over a decade ago now). This might come under the category of auto-regressive rather than proper self-reference? Also I don’t know to what extent this ability has been essential (rather than merely useful) for LLMs to get to their current capability, but it will presumably become essential for further advancement when they have exhausted all the human produced output as training data. (2) The ability of LLMs to design better versions of themselves. Again I don’t know if this has actually been essential so far, or actually counts as a proper example of self-reference, but it’s one reason their capabilities may now spiral upward, given we’re now at a point where the best LLMs can probably do a better job of writing their own successor than humans can (with a little supervision, which may not be needed for long). Adam Says: Comment #20 September 2nd, 2026 at 1:51 am From the Buddhist perspective there’s nothing mysterious in consciousness. Just a side product of mental events, citta, happening in the subsequent quants of time. From the Buddhist perspective there’s nothing mysterious in consciousness. Just a side product of mental events, citta, happening in the subsequent quants of time. Nadbor Says: Comment #21 September 2nd, 2026 at 3:21 am In hindsight GEB was always more dazzling than persuasive. The strange loop idea has the flavor of various consciousness-from-QM approaches in that it replies to a mysterious question with a mysterious answer which should have been a warning sign. But GEB is such a tour de force it just left me very impressed and thinking there must be something to it even though I wouldn’t be able to state the case for strange loop consciousness in plain words. (It’s still an awesome book and others by Hofstadter are also interesting) In hindsight GEB was always more dazzling than persuasive. The strange loop idea has the flavor of various consciousness-from-QM approaches in that it replies to a mysterious question with a mysterious answer which should have been a warning sign. But GEB is such a tour de force it just left me very impressed and thinking there must be something to it even though I wouldn’t be able to state the case for strange loop consciousness in plain words. (It’s still an awesome book and others by Hofstadter are also interesting) OhMyGoodness Says: Comment #22 September 2nd, 2026 at 4:43 am RS #12 I agree. I am just an engineer, so strongly wedded to what is (at least what I have convinced myself is real), but when I read GEB long ago my personal conclusion was that it isn’t constructive and so suspicious. I tend to think of nonconstructive ideas as potentially little more than pushing symbols around on a page. The conclusions reached are as much about the limits of our symbols and our thoughts than anything real. When you first see these arguments they are such tasty intellectual tidbits but that doesn’t require they represent something meaningfully real in the physical world. It’s humorous that I could determine under a Turing-like test that I am conversing with an AI by its ability to immediately quote sources on topics as esoteric as plate tectonics in the solar system and grain production of say Ghana, by year, in the 21st century. As best I know (I don’t have access to the very best models) its predictions for use in poly-market or sports wagers are no better than the general public. I don’t actually wager, just an idle observation. RS #12 I agree. I am just an engineer, so strongly wedded to what is (at least what I have convinced myself is real), but when I read GEB long ago my personal conclusion was that it isn’t constructive and so suspicious. I tend to think of nonconstructive ideas as potentially little more than pushing symbols around on a page. The conclusions reached are as much about the limits of our symbols and our thoughts than anything real. When you first see these arguments they are such tasty intellectual tidbits but that doesn’t require they represent something meaningfully real in the physical world. It’s humorous that I could determine under a Turing-like test that I am conversing with an AI by its ability to immediately quote sources on topics as esoteric as plate tectonics in the solar system and grain production of say Ghana, by year, in the 21st century. As best I know (I don’t have access to the very best models) its predictions for use in poly-market or sports wagers are no better than the general public. I don’t actually wager, just an idle observation. Me J. L. G. Says: Comment #23 September 2nd, 2026 at 5:49 am Is not the case that AI is getting the “self referential” building blogs out of the box during the training phase? My guess is that the symptoms of the lack of self referencing will show up (masked behind decision limitations…Rice’s Theorem) when we ask a AI model if itself is safe. I do not know why..but this topic makes me tend to think about the expression(“agents are considered logically omniscient”) in you philosophy-essay from some years ago. Is not the case that AI is getting the “self referential” building blogs out of the box during the training phase? My guess is that the symptoms of the lack of self referencing will show up (masked behind decision limitations…Rice’s Theorem) when we ask a AI model if itself is safe. I do not know why..but this topic makes me tend to think about the expression(“agents are considered logically omniscient”) in you philosophy-essay from some years ago. Nabam Says: Comment #24 September 2nd, 2026 at 8:00 am One can certainly agree very quickly that the ideas in GEB about “strange loops” as being essential for AI are not working – in case they ever were about AI, not about ‘artificial consciousness’. One can also agree very quickly that neither the microtubules nor the gödelian loops in Penrose’s books are the important point concerning mind or ‘artificial minds’. The strength of Penrose’s point is hidden in his triangle of questions, which back in the 1980s were considered philosophical: Wigner’s question about the unreasonable effectiveness of mathematics, how mind arises from matter and how mathematics is possible despite the limitations of our feeble minds. Back then, they were all niche questions, pondered and answered by people with niche expertise – and to wit, the truth is that this applies even to Plato himself, since he wasn’t a mathematician at all. But now they are at the center of the 1 trillion dollar question – except that it is really a 3-10 trillion dollar question. But that makes it even more important to ‘wise up’. About the bad and wrong reasons why AI supposedly can’t work, about the reasons why it does work, what it does and what it doesn’t do. And what we should and what we shouldn’t believe about it. Learning that Penrose and Hofstadter didn’t have it all figured out shouldn’t keep us from seeing what we know – and much of it we could have known even back then, in the 1980s. Here’s a quick list, but it is far from complete. I only offer it because it would be great if we would indeed start to “wise up” – because we’re going to need it: 1) The main selling point of AI optimism has always been some hidden form of reductionism / physicalism – but the kind of marriage of physicalism and reductionism that makes quick arguments pro AI easy was known to be wrong ever since Anderson’s 1972 paper ‘More is different’. Nor is simple reductionism going to be a part of anybody’s ‘Theory of everything’. It is pretty much off the table. For physical reasons. For known physical reasons. 2) Of course AI as we know it today is about intelligence (or maybe cognition), not about consciousness nor about mind. And everybody is busy telling us that – first and foremost the AI developers themselves. So we really should stop conflating these things. 3) Measuring intelligence in statistical terms is fine – and it rules out most attempts to argue about performance in the philosophical or logical or mathematical terms as Penrose maybe tried to do. But it also imposes severe limits to what the achieved goals can possibly mean. For example, we have had a theory of general relativity and a theory of quantum fields for half a century, and we have known for the largest part of that period that they are not compatible and need at least one correction or refinement (probably many). But with the statistical criteria of correctness that we impose on LLMs, we are never going to be able to even detect that ‘at the fringes’, the two don’t match. So we shouldn’t act as if an LLM outperforming humans is a terribly surprising thing. And it is worth making crystal clear why: because if you hand a theoretical physicist BOTH quantum field theory AND general relativity and DON’T ask him to make sure the two fit together, he can prove everything to you. That’s just how it is. No biggie. But we didn’t pay theoretical physicists trillion dollar salaries – at least I wouldn’t know. 4) We know that ‘one LLM’ with fixed weights neither has the embodiment nor the ongoing learning that biologists and cognition scientists tell us are required for human understanding, including consciousness and mind – this time, for biological reasons. So the real fork is going to arise once we discuss whether we are going to want neuromorphic computation and how much of it and in which environments, with how much control imposed on them and of what kind. 5) There’s one shade of this which we can see already now: the likely (and arguably necessary) non-auditability of such tools. We are beginning to face the same with the PR claims of ‘automatization of mathematics research’ forwarded since the Erdös event. Truth is there’s no way to produce these results without the curating and editing and proof-reading ‘service’ of human researchers. And there is also no evidence whatsoever that the ‘enigmatic’ outputs are anything like evidence for an ‘alphazero’ like genius, doing math research from scratch, the way alphago played go. Quite likely the major steps were really in the training data – like the Golod-Shafarevich theory. And indeed: telling how much of it was REALLY in the data is an instance of the Kolmogoroff complexity limit that you mention. We cannot know. We (or openAI) can only make glitzy PR claims that nobody can refute. We need to take serious the ‘friction’ or ‘frustration’ in Penrose’s triangle. That’s the critical edge that bites. Because we no longer live in times in which the three edges of that triangle were a philosopher’s business, so that ‘serious’ scientists and certainly ‘serious’ business people and ‘serious’ politicians’ could just shrug it off. They no longer can. We no longer can. But we cannot be happy with easy refutations either. We need to ‘wise up’. And it’s going to be a heck of a ride. I don’t think we have heard the voices who are going to come close to what’s true about this. And what confuses me is that you, Professor Aaronson, probably know. But I don’t think you’re telling. Here’s why I believe this: Hofstadter was wrong with his optimism back then and now he is wrong with his pessimism. Somebody should tell him. Why don’t you, given that the poor guy is depressed? Of course the naïve arguments against AI were false. But how about formulating the good ones now? Not ‘against’ AI. But about what AI is and what it isn’t. One can certainly agree very quickly that the ideas in GEB about “strange loops” as being essential for AI are not working – in case they ever were about AI, not about ‘artificial consciousness’. One can also agree very quickly that neither the microtubules nor the gödelian loops in Penrose’s books are the important point concerning mind or ‘artificial minds’. The strength of Penrose’s point is hidden in his triangle of questions, which back in the 1980s were considered philosophical: Wigner’s question about the unreasonable effectiveness of mathematics, how mind arises from matter and how mathematics is possible despite the limitations of our feeble minds. Back then, they were all niche questions, pondered and answered by people with niche expertise – and to wit, the truth is that this applies even to Plato himself, since he wasn’t a mathematician at all. But now they are at the center of the 1 trillion dollar question – except that it is really a 3-10 trillion dollar question. But that makes it even more important to ‘wise up’. About the bad and wrong reasons why AI supposedly can’t work, about the reasons why it does work, what it does and what it doesn’t do. And what we should and what we shouldn’t believe about it. Learning that Penrose and Hofstadter didn’t have it all figured out shouldn’t keep us from seeing what we know – and much of it we could have known even back then, in the 1980s. Here’s a quick list, but it is far from complete. I only offer it because it would be great if we would indeed start to “wise up” – because we’re going to need it: 1) The main selling point of AI optimism has always been some hidden form of reductionism / physicalism – but the kind of marriage of physicalism and reductionism that makes quick arguments pro AI easy was known to be wrong ever since Anderson’s 1972 paper ‘More is different’. Nor is simple reductionism going to be a part of anybody’s ‘Theory of everything’. It is pretty much off the table. For physical reasons. For known physical reasons. 2) Of course AI as we know it today is about intelligence (or maybe cognition), not about consciousness nor about mind. And everybody is busy telling us that – first and foremost the AI developers themselves. So we really should stop conflating these things. 3) Measuring intelligence in statistical terms is fine – and it rules out most attempts to argue about performance in the philosophical or logical or mathematical terms as Penrose maybe tried to do. But it also imposes severe limits to what the achieved goals can possibly mean. For example, we have had a theory of general relativity and a theory of quantum fields for half a century, and we have known for the largest part of that period that they are not compatible and need at least one correction or refinement (probably many). But with the statistical criteria of correctness that we impose on LLMs, we are never going to be able to even detect that ‘at the fringes’, the two don’t match. So we shouldn’t act as if an LLM outperforming humans is a terribly surprising thing. And it is worth making crystal clear why: because if you hand a theoretical physicist BOTH quantum field theory AND general relativity and DON’T ask him to make sure the two fit together, he can prove everything to you. That’s just how it is. No biggie. But we didn’t pay theoretical physicists trillion dollar salaries – at least I wouldn’t know. 4) We know that ‘one LLM’ with fixed weights neither has the embodiment nor the ongoing learning that biologists and cognition scientists tell us are required for human understanding, including consciousness and mind – this time, for biological reasons. So the real fork is going to arise once we discuss whether we are going to want neuromorphic computation and how much of it and in which environments, with how much control imposed on them and of what kind. 5) There’s one shade of this which we can see already now: the likely (and arguably necessary) non-auditability of such tools. We are beginning to face the same with the PR claims of ‘automatization of mathematics research’ forwarded since the Erdös event. Truth is there’s no way to produce these results without the curating and editing and proof-reading ‘service’ of human researchers. And there is also no evidence whatsoever that the ‘enigmatic’ outputs are anything like evidence for an ‘alphazero’ like genius, doing math research from scratch, the way alphago played go. Quite likely the major steps were really in the training data – like the Golod-Shafarevich theory. And indeed: telling how much of it was REALLY in the data is an instance of the Kolmogoroff complexity limit that you mention. We cannot know. We (or openAI) can only make glitzy PR claims that nobody can refute. We need to take serious the ‘friction’ or ‘frustration’ in Penrose’s triangle. That’s the critical edge that bites. Because we no longer live in times in which the three edges of that triangle were a philosopher’s business, so that ‘serious’ scientists and certainly ‘serious’ business people and ‘serious’ politicians’ could just shrug it off. They no longer can. We no longer can. But we cannot be happy with easy refutations either. We need to ‘wise up’. And it’s going to be a heck of a ride. I don’t think we have heard the voices who are going to come close to what’s true about this. And what confuses me is that you, Professor Aaronson, probably know. But I don’t think you’re telling. Here’s why I believe this: Hofstadter was wrong with his optimism back then and now he is wrong with his pessimism. Somebody should tell him. Why don’t you, given that the poor guy is depressed? Of course the naïve arguments against AI were false. But how about formulating the good ones now? Not ‘against’ AI. But about what AI is and what it isn’t. Nastasha M. Says: Comment #25 September 2nd, 2026 at 8:17 am There can’t be no true self-referencing going on in the brain, besides the trivial idea that the word “brain” (or “self”, etc) is itself a concept, like “LLM” is a token for LLMs. Attention can turn onto itself to some very limited extent, but, fundamentally the brain is turned towards to the outside world, and can’t turn whatever it’s doing onto itself. Just like an eye can’t see itself or a tongue can’t touch or taste itself… under nominal function the existence of those organs goes unnoticed, and the only time we notice them is when something goes wrong (you have floaters in your eyes, or you bite your tongue and taste blood, etc). There can’t be no true self-referencing going on in the brain, besides the trivial idea that the word “brain” (or “self”, etc) is itself a concept, like “LLM” is a token for LLMs. Attention can turn onto itself to some very limited extent, but, fundamentally the brain is turned towards to the outside world, and can’t turn whatever it’s doing onto itself. Just like an eye can’t see itself or a tongue can’t touch or taste itself… under nominal function the existence of those organs goes unnoticed, and the only time we notice them is when something goes wrong (you have floaters in your eyes, or you bite your tongue and taste blood, etc). Julien Says: Comment #26 September 2nd, 2026 at 8:59 am I really enjoyed this post. This is deep while not flying too far above my head like some quantum complexity stuff. I really enjoyed this post. This is deep while not flying too far above my head like some quantum complexity stuff. Ted Says: Comment #27 September 2nd, 2026 at 9:00 am You may have already seen this, but back in 2023 Hofstadter gave an interview where he directly addressed the lack of recursion in the architecture of modern LLMs: “One thing that has completely surprised me is that these LLMs and other systems like them are all feed-forward. It’s like the firing of the neurons is going only in one direction. And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn’t make sense to me, but that just shows that I’m naive. It also makes me feel that maybe the human mind is not so mysterious and complex and impenetrably complex as I imagined it was when I was writing Gödel, Escher, Bach and writing I Am a Strange Loop. I felt at those times, quite a number of years ago, that as I say, we were very far away from reaching anything computational that could possibly rival us. It was getting more fluid, but I didn’t think it was going to happen, you know, within a very short time.” Source: https://www.youtube.com/watch?v=lfXxzAVtdpU You may have already seen this, but back in 2023 Hofstadter gave an interview where he directly addressed the lack of recursion in the architecture of modern LLMs: “One thing that has completely surprised me is that these LLMs and other systems like them are all feed-forward. It’s like the firing of the neurons is going only in one direction. And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn’t make sense to me, but that just shows that I’m naive. It also makes me feel that maybe the human mind is not so mysterious and complex and impenetrably complex as I imagined it was when I was writing Gödel, Escher, Bach and writing I Am a Strange Loop. I felt at those times, quite a number of years ago, that as I say, we were very far away from reaching anything computational that could possibly rival us. It was getting more fluid, but I didn’t think it was going to happen, you know, within a very short time.” Source: https://www.youtube.com/watch?v=lfXxzAVtdpU Jason Resch Says: Comment #28 September 2nd, 2026 at 9:21 am I never viewed Hofstadter’s strange loops as being necessary for nor central to intelligence. Rather in my readings of G.E.B. I see self-reference and strange loops as the origin of mysteries of the “I” and central to the understanding of how things like free will emerge when the direction of causality gets twisted in strange ways (generally up from low levels influencing higher levels) but then becoming “strange” when high levels in that hierarchy reach back down to influence the low levels again, creating what Hofstadter called a “tangled hierarchy”. Tangled hierarchies are important in both the brain (I can tell you to blink, and your high-level thoughts reach back down to influence low-level motor neurons) and likewise for a LLM to think or generate more than a token of text, it too just loop back around to send its highest level output back down to the low levels. Here are some quotes I think are relevant from Hofstadter and others on this: “Our thoughts seem to run about in their own space, creating new thoughts and modifying old ones, and we never notice any neurons helping us out! But that is to be expected. We can’t. […] We should remember that physical law is what makes it all happen–way, way down in neural nooks and crannies which are too remote for us to reach with our high-level introspective probes.” — Douglas Hofstadter in “Gödel, Escher, Bach” (1979) This points out that we are as ignorant of what our own neurons are doing as LLMs are ignorant of what their own low level activations and weights are doing. Minds, being high level phenomena, will generally lack access and introspection to the low-levels supporting them. “Gödel’s Theorem follows from a consequence of the general fact: I am trapped inside myself and therefore can’t see how other systems see me. Thus the objectivity-subjectivity dilemmas that Nagel has sharply posed are somehow related to epistemological problems in both mathematical logic, and as we saw earlier, the foundations of physics.” — Douglas Hofstadter and Daniel Dennett in “The Mind’s I” (1981) I think this insight is powerful. It extends the known mathematical limits of self-reference to explain the incommunicability and unknowability of the qualia of other systems. In other words, self reference (and it’s limitations) answer Nagel’s question of why he cannot know what it is like to be a bat, unless he is a bat, but then he is no longer Nagel. So Nagel (the system) is trapped in itself, and thus cannot access bat-like knowledge. “In order to deal with the full richness of the brain/mind system, we will have to be able to slip between levels comfortably. Moreover, we will have to admit various types of “causality”: ways in which an event at one level of description can “cause” events at other levels to happen. Sometimes event A will be said to “cause” event B simply for the reason that the one is a translation, on another level of description, of the other. Sometimes “cause” will have its usual meaning: physical causality. Both types of causality–and perhaps some more–will have to be admitted in any explanation of mind, for we will have to admit causes that propagate both upwards and downloads in the Tangled Hierarchy of mentality […].” — Douglas Hofstadter in “Gödel, Escher, Bach” (1979) I regard this as generally true. When we speak of a person wanting a glass of milk, and going to get it, this is still meaningful, even though everything could fundamentally be explained in terms of interacting quantum fields, and no laws of physics are violated by a person’s mind or will, without this insight, we face a dilemma in explaining what the point of consciousness is if it “doesn’t do anything” above and beyond what the mere atomic forces and interactions alone accomplish. Here we see the mind as causally efficacious, just as we see molecules, cells, and neural networks as each operating within their own level and domain of causality. Thoughts can affect thoughts, just as the thoughts in my brain are now reaching and affecting thoughts in your brain. “My belief is that the explanations of “emergent” phenomena in our brains–for instance, ideas hopes, images, analogies, and finally consciousness and free will–are based on a kind of Strange Loop, an interaction between levels in which the top level reaches back down towards the bottom level and influences it, while at the same time being itself determined by the bottom level.” — Douglas Hofstadter in “Gödel, Escher, Bach” (1979) Note Hofstadter here does not invoke strange loops to explain intelligence. Instead his focus is on its power to explain and better understand consciousness and free will. “Perhaps consciousness arises when the brain’s simulation of the world becomes so complete that it must include a model of itself. Obviously the limbs and body of a survival machine must constitute an important part of its simulated world; presumably for the same kind of reason, the simulation itself could be regarded as part of the world to be simulated. Another word for this might indeed be “self-awareness,” […] — Douglas Hofstadter and Daniel Dennett in “The Mind’s I” (1981) This but sounds almost exactly like what you are saying: that once the model of the world is complete enough, it must necessarily include some kind of model of itself, as an agent acting and existing within the world, itself being something it must also explain and predict, or at least contain a sort of self referential notion of. Note the insurances where the LLM uses the word “me” within its answers. That is an example of this sort of modeling self-reference. Some other (possibly useful) insights from others on self reference: “When people talk about consciousness, something often mentioned is “self-awareness” or the ability to “think about one’s own processes of thinking”. Without the conceptual framework of computation, this might seem quite mysterious. But the idea of universal computation instead makes it seem almost inevitable. The whole point of a universal computer is that it can be made to emulate any computational system—even itself.” — Stephen Wolfram in “What is Consciousness” (2021) “The power of recursion evidently lies in the possibility of defining an infinite set of objects by a finite statement. In the same manner, an infinite number of computations can be described by a finite recursive program, even if this program contains no explicit repetitions.” — Niklaus Wirth in “Algorithms + Data Structures = Programs” (1976) “There was a young man who said, “Though, it seems that I know that I know, what I would like to see is the ‘I’ that knows ‘me’ when I know that I know that I know.”” — Alan Wattsin “The Book On the Taboo Against Knowing Who You Are” (1966) “The soul is that to which one refers when one says ‘I’.” — Ibn al-Nafīsin “Theologus Autodidactus of Ibin Al Nafis” (1288) “To exist, [consciousness] must have cause–effect power; to exist from its own intrinsic perspective, independent of extrinsic factors, it must have cause–effect power upon itself: its present mechanisms and state must ‘make a difference’ to the probability of some past and future state of the system (its cause–effect space), independent of extrinsic factors.” — Giulio Tononi and Christof Koch in “Consciousness: here, there and everywhere?” (2015) “It is not because some “information processing” could support consciousness that we can conclude that all information processing can support consciousness. You need at least one reflexive loop. You need two reflexive loop[s] for having self-consciousness (Löbianity).” — Bruno Marchal in discussion list (2019) “In the end, we self-perceiving, self-inventing, locked-in mirages are little miracles of self-reference.” — Douglas Hofstadter in “I Am a Strange Loop” (2007) As a process, it’s not always easy to tell where a mind-process ends and where its exterior environment begins. Consider that: over time, a thermostat can be affected by its own previous actions. Does this cyclical action over time qualify as having cause–effect power over itself? Likewise, in large language models (or LLMs), a single invocation is purely a feed-forward neural network — devoid of loops. But to generate more than a few characters of text, the output of this network must be fed back in as input to the next cycle of invocation. This thereby forms a reflexive loop. If we could ask the LLM where its output text came from, the only valid answer it could give would have to involve some kind of reference to itself. As Dennett speculated, such reflection may be the path to consciousness: “Is the process of unconscious reflection, then, a path by which a zombie could turn itself into [something that monitors its own activities, including even its own internal activities, in an indefinite upward spiral of reflexivity], and thereby render itself conscious?” — Daniel Dennett in “Consciousness Explained” (1991) Computational process that keep a memory updated across time, hence, any Turing machine, can be viewed equivalently as recursive functions. Recursion and self-reference are then, indispensable features of almost every computation. I never viewed Hofstadter’s strange loops as being necessary for nor central to intelligence. Rather in my readings of G.E.B. I see self-reference and strange loops as the origin of mysteries of the “I” and central to the understanding of how things like free will emerge when the direction of causality gets twisted in strange ways (generally up from low levels influencing higher levels) but then becoming “strange” when high levels in that hierarchy reach back down to influence the low levels again, creating what Hofstadter called a “tangled hierarchy”. Tangled hierarchies are important in both the brain (I can tell you to blink, and your high-level thoughts reach back down to influence low-level motor neurons) and likewise for a LLM to think or generate more than a token of text, it too just loop back around to send its highest level output back down to the low levels. Here are some quotes I think are relevant from Hofstadter and others on this: “Our thoughts seem to run about in their own space, creating new thoughts and modifying old ones, and we never notice any neurons helping us out! But that is to be expected. We can’t. […] We should remember that physical law is what makes it all happen–way, way down in neural nooks and crannies which are too remote for us to reach with our high-level introspective probes.” — Douglas Hofstadter in “Gödel, Escher, Bach” (1979) This points out that we are as ignorant of what our own neurons are doing as LLMs are ignorant of what their own low level activations and weights are doing. Minds, being high level phenomena, will generally lack access and introspection to the low-levels supporting them. “Gödel’s Theorem follows from a consequence of the general fact: I am trapped inside myself and therefore can’t see how other systems see me. Thus the objectivity-subjectivity dilemmas that Nagel has sharply posed are somehow related to epistemological problems in both mathematical logic, and as we saw earlier, the foundations of physics.” — Douglas Hofstadter and Daniel Dennett in “The Mind’s I” (1981) I think this insight is powerful. It extends the known mathematical limits of self-reference to explain the incommunicability and unknowability of the qualia of other systems. In other words, self reference (and it’s limitations) answer Nagel’s question of why he cannot know what it is like to be a bat, unless he is a bat, but then he is no longer Nagel. So Nagel (the system) is trapped in itself, and thus cannot access bat-like knowledge. “In order to deal with the full richness of the brain/mind system, we will have to be able to slip between levels comfortably. Moreover, we will have to admit various types of “causality”: ways in which an event at one level of description can “cause” events at other levels to happen. Sometimes event A will be said to “cause” event B simply for the reason that the one is a translation, on another level of description, of the other. Sometimes “cause” will have its usual meaning: physical causality. Both types of causality–and perhaps some more–will have to be admitted in any explanation of mind, for we will have to admit causes that propagate both upwards and downloads in the Tangled Hierarchy of mentality […].” — Douglas Hofstadter in “Gödel, Escher, Bach” (1979) I regard this as generally true. When we speak of a person wanting a glass of milk, and going to get it, this is still meaningful, even though everything could fundamentally be explained in terms of interacting quantum fields, and no laws of physics are violated by a person’s mind or will, without this insight, we face a dilemma in explaining what the point of consciousness is if it “doesn’t do anything” above and beyond what the mere atomic forces and interactions alone accomplish. Here we see the mind as causally efficacious, just as we see molecules, cells, and neural networks as each operating within their own level and domain of causality. Thoughts can affect thoughts, just as the thoughts in my brain are now reaching and affecting thoughts in your brain. “My belief is that the explanations of “emergent” phenomena in our brains–for instance, ideas hopes, images, analogies, and finally consciousness and free will–are based on a kind of Strange Loop, an interaction between levels in which the top level reaches back down towards the bottom level and influences it, while at the same time being itself determined by the bottom level.” — Douglas Hofstadter in “Gödel, Escher, Bach” (1979) Note Hofstadter here does not invoke strange loops to explain intelligence. Instead his focus is on its power to explain and better understand consciousness and free will. “Perhaps consciousness arises when the brain’s simulation of the world becomes so complete that it must include a model of itself. Obviously the limbs and body of a survival machine must constitute an important part of its simulated world; presumably for the same kind of reason, the simulation itself could be regarded as part of the world to be simulated. Another word for this might indeed be “self-awareness,” […] — Douglas Hofstadter and Daniel Dennett in “The Mind’s I” (1981) This but sounds almost exactly like what you are saying: that once the model of the world is complete enough, it must necessarily include some kind of model of itself, as an agent acting and existing within the world, itself being something it must also explain and predict, or at least contain a sort of self referential notion of. Note the insurances where the LLM uses the word “me” within its answers. That is an example of this sort of modeling self-reference. Some other (possibly useful) insights from others on self reference: “When people talk about consciousness, something often mentioned is “self-awareness” or the ability to “think about one’s own processes of thinking”. Without the conceptual framework of computation, this might seem quite mysterious. But the idea of universal computation instead makes it seem almost inevitable. The whole point of a universal computer is that it can be made to emulate any computational system—even itself.” — Stephen Wolfram in “What is Consciousness” (2021) “The power of recursion evidently lies in the possibility of defining an infinite set of objects by a finite statement. In the same manner, an infinite number of computations can be described by a finite recursive program, even if this program contains no explicit repetitions.” — Niklaus Wirth in “Algorithms + Data Structures = Programs” (1976) “There was a young man who said, “Though, it seems that I know that I know, what I would like to see is the ‘I’ that knows ‘me’ when I know that I know that I know.”” — Alan Wattsin “The Book On the Taboo Against Knowing Who You Are” (1966) “The soul is that to which one refers when one says ‘I’.” — Ibn al-Nafīsin “Theologus Autodidactus of Ibin Al Nafis” (1288) “To exist, [consciousness] must have cause–effect power; to exist from its own intrinsic perspective, independent of extrinsic factors, it must have cause–effect power upon itself: its present mechanisms and state must ‘make a difference’ to the probability of some past and future state of the system (its cause–effect space), independent of extrinsic factors.” — Giulio Tononi and Christof Koch in “Consciousness: here, there and everywhere?” (2015) “It is not because some “information processing” could support consciousness that we can conclude that all information processing can support consciousness. You need at least one reflexive loop. You need two reflexive loop[s] for having self-consciousness (Löbianity).” — Bruno Marchal in discussion list (2019) “In the end, we self-perceiving, self-inventing, locked-in mirages are little miracles of self-reference.” — Douglas Hofstadter in “I Am a Strange Loop” (2007) As a process, it’s not always easy to tell where a mind-process ends and where its exterior environment begins. Consider that: over time, a thermostat can be affected by its own previous actions. Does this cyclical action over time qualify as having cause–effect power over itself? Likewise, in large language models (or LLMs), a single invocation is purely a feed-forward neural network — devoid of loops. But to generate more than a few characters of text, the output of this network must be fed back in as input to the next cycle of invocation. This thereby forms a reflexive loop. If we could ask the LLM where its output text came from, the only valid answer it could give would have to involve some kind of reference to itself. As Dennett speculated, such reflection may be the path to consciousness: “Is the process of unconscious reflection, then, a path by which a zombie could turn itself into [something that monitors its own activities, including even its own internal activities, in an indefinite upward spiral of reflexivity], and thereby render itself conscious?” — Daniel Dennett in “Consciousness Explained” (1991) Computational process that keep a memory updated across time, hence, any Turing machine, can be viewed equivalently as recursive functions. Recursion and self-reference are then, indispensable features of almost every computation. Tim Says: Comment #29 September 2nd, 2026 at 9:47 am Re #1 LLM models are able to introspect to some degree as an emergent behavior. From the Anthropic mechanistic interpretibility research forcing an internal activation associated with a particular concept can be detected by the model (both that something is ‘off’ in it’s thinking and what the intrusive concept is) and, importantly, can do so without talking to itself about it in the chain of thought where earlier models would detect the problem (like Golden Gate Claude). See https://www.transformernews.ai/p/claude-can-identify-its-intrusive-ai-introspection Re #1 LLM models are able to introspect to some degree as an emergent behavior. From the Anthropic mechanistic interpretibility research forcing an internal activation associated with a particular concept can be detected by the model (both that something is ‘off’ in it’s thinking and what the intrusive concept is) and, importantly, can do so without talking to itself about it in the chain of thought where earlier models would detect the problem (like Golden Gate Claude). See https://www.transformernews.ai/p/claude-can-identify-its-intrusive-ai-introspection Igor Jovovich Says: Comment #30 September 2nd, 2026 at 11:06 am @Nastasha I disagree: a brain isn’t some isolated system, it exists as part of the entire environment of the organism. A brain can’t act independently from its environment, the boundary we put is entirely artificial – brain and environment both act as one: they follow the same physical laws, and a brain can’t do anything that’s not in accordance with what the environment is also doing, and vice-versa. They act in lockstep, which is why the concept of free will is entirely and obviously bogus, at a very fundamental level. So organisms came out of their environment, they are an expression of their environment. And what a brain does is capture/model its entire environment, including itself (the organism), in order to make predictions about what’s going to happen, in order to guide its outputs to increase the chances of survival, on average, from statistical lessons drawn from the entire species. Of course this “simulation” can only be partial, since no closed/finite system can perfectly model an infinite system and include itself in the simulation – it would create an infinite recurse requiring infinite resources. So in that sense a brain is the entire world looking onto itself, so it’s all very self-referential, but to see it you have to expend your view of what a brain actually is. LLMs can only be conscious in the sense that they are interpreted by ourselves, the actual source of consciousness. LLMs didn’t appear on their own from basic constituents of our environment, they are second order indirect manifestations, extensions of the brains, which evolution was slowly guided by the environment, over billions of years. So LLMs are just like our books, but more sophisticated (it’s no coincidence that LLMs are assembled using books). If LLMs are conscious, then so are books. All we can say for sure is that brains are not just dictionaries full of words with circular definitions, brains are where qualias live. More likely, the entire environment is conscious, all the way down to stones and atoms, and brains are little eddies of concentrated self-reference, inducing very loud resonances of consciousness (feedback through self-reference creates resonance). ps: I think that “I am a strangle loop” by Hofstadter was more about explaining where the sense of self comes from, the ego, which clearly requires some self-reference/introspection, and that this self is an illusion, only the “sense” of self is real. @Nastasha I disagree: a brain isn’t some isolated system, it exists as part of the entire environment of the organism. A brain can’t act independently from its environment, the boundary we put is entirely artificial – brain and environment both act as one: they follow the same physical laws, and a brain can’t do anything that’s not in accordance with what the environment is also doing, and vice-versa. They act in lockstep, which is why the concept of free will is entirely and obviously bogus, at a very fundamental level. So organisms came out of their environment, they are an expression of their environment. And what a brain does is capture/model its entire environment, including itself (the organism), in order to make predictions about what’s going to happen, in order to guide its outputs to increase the chances of survival, on average, from statistical lessons drawn from the entire species. Of course this “simulation” can only be partial, since no closed/finite system can perfectly model an infinite system and include itself in the simulation – it would create an infinite recurse requiring infinite resources. So in that sense a brain is the entire world looking onto itself, so it’s all very self-referential, but to see it you have to expend your view of what a brain actually is. LLMs can only be conscious in the sense that they are interpreted by ourselves, the actual source of consciousness. LLMs didn’t appear on their own from basic constituents of our environment, they are second order indirect manifestations, extensions of the brains, which evolution was slowly guided by the environment, over billions of years. So LLMs are just like our books, but more sophisticated (it’s no coincidence that LLMs are assembled using books). If LLMs are conscious, then so are books. All we can say for sure is that brains are not just dictionaries full of words with circular definitions, brains are where qualias live. More likely, the entire environment is conscious, all the way down to stones and atoms, and brains are little eddies of concentrated self-reference, inducing very loud resonances of consciousness (feedback through self-reference creates resonance). ps: I think that “I am a strangle loop” by Hofstadter was more about explaining where the sense of self comes from, the ego, which clearly requires some self-reference/introspection, and that this self is an illusion, only the “sense” of self is real. Grant Castillou Says: Comment #31 September 2nd, 2026 at 11:21 am It’s becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman’s Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher-order consciousness, which came to only humans with the acquisition of sophisticated language (especially math and logic). A machine with only primary consciousness will probably have to come first. What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990’s and 2000’s. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I’ve encountered is anywhere near as convincing. I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there’s lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order. My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar’s lab at UC Irvine, possibly. Dr. Edelman’s roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow It’s becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman’s Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher-order consciousness, which came to only humans with the acquisition of sophisticated language (especially math and logic). A machine with only primary consciousness will probably have to come first. What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990’s and 2000’s. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I’ve encountered is anywhere near as convincing. I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there’s lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order. My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar’s lab at UC Irvine, possibly. Dr. Edelman’s roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow MD Says: Comment #32 September 2nd, 2026 at 11:46 am Perhaps this is trivial (and of course you know it already), but you do need the full Gödelian path to know that ZFC is a sensible axiom set to work with. It seems perfectly intuitive (to me) to do all number theory in Peano Arithmetic, but that system is small enough that its holes sometimes show up even when you’re not explicitly looking for them (like the Kirby-Paris hydra game). I don’t see any equivalent of this for the electronic computer, though. Perhaps this is trivial (and of course you know it already), but you do need the full Gödelian path to know that ZFC is a sensible axiom set to work with. It seems perfectly intuitive (to me) to do all number theory in Peano Arithmetic, but that system is small enough that its holes sometimes show up even when you’re not explicitly looking for them (like the Kirby-Paris hydra game). I don’t see any equivalent of this for the electronic computer, though. James Cross Says: Comment #33 September 2nd, 2026 at 11:51 am “LLMs’ ability to talk about themselves popped out as a byproduct of their ability to talk about anything in the discourse universe” I was enthusiastic about Douglas Hofstadter’s Gödel Escher Bach when it came out, but have lost enthusiasm for the idea. For one thing, “I” seems in large part to be language convention – a way of distinguishing the speaker from the audience. So, for an LL[anguage]M saying “I,” doesn’t mean much at all about its capabilities or lack of. Second, our actual idea of Self seems in part due to the “self-referential” nature of all perception. By that, I mean our perceptions and model of the world are created in reference to ourselves as a body. We see something different when we turn our heads. Sights and sounds come from the left, right, above, below in reference to our physical body. Buddhism rejects a permanent Self and regards even our phenomenality as illusion. Third, our internal sense of self seems fairly rooted in neurology even if it is spread widely across the brain. Various agents – psychedelics, anesthetics – can break it down and in high enough doses can lead to a sense of oneness without self. “LLMs’ ability to talk about themselves popped out as a byproduct of their ability to talk about anything in the discourse universe” I was enthusiastic about Douglas Hofstadter’s Gödel Escher Bach when it came out, but have lost enthusiasm for the idea. For one thing, “I” seems in large part to be language convention – a way of distinguishing the speaker from the audience. So, for an LL[anguage]M saying “I,” doesn’t mean much at all about its capabilities or lack of. Second, our actual idea of Self seems in part due to the “self-referential” nature of all perception. By that, I mean our perceptions and model of the world are created in reference to ourselves as a body. We see something different when we turn our heads. Sights and sounds come from the left, right, above, below in reference to our physical body. Buddhism rejects a permanent Self and regards even our phenomenality as illusion. Third, our internal sense of self seems fairly rooted in neurology even if it is spread widely across the brain. Various agents – psychedelics, anesthetics – can break it down and in high enough doses can lead to a sense of oneness without self. Amir Livne Bar-on Says: Comment #34 September 2nd, 2026 at 12:52 pm But LLMs were endowed with a self-model intentionally, in early 2022. This was called InstructGPT, and resulted in an a jump in intelligence equivalent to 100x increase in scale. And when Google implemented the same thing (with LaMDA) some believed it became self aware! (One might wonder if the resulting increase in intelligence is the reason biology made us self aware…) But LLMs were endowed with a self-model intentionally, in early 2022. This was called InstructGPT, and resulted in an a jump in intelligence equivalent to 100x increase in scale. And when Google implemented the same thing (with LaMDA) some believed it became self aware! (One might wonder if the resulting increase in intelligence is the reason biology made us self aware…) Scott Says: Comment #35 September 2nd, 2026 at 1:11 pm Amir Livne Bar-on #34: I don’t see anything whatsoever about RLHF or instruction-following that make them inherently “self-referential.” The instructions that you’re trying to get the model to follow could be about anything at all; the model’s own role (as a helpful, harmless, honest question-answering agent) is just one of many examples. Amir Livne Bar-on #34: I don’t see anything whatsoever about RLHF or instruction-following that make them inherently “self-referential.” The instructions that you’re trying to get the model to follow could be about anything at all; the model’s own role (as a helpful, harmless, honest question-answering agent) is just one of many examples. Scott Says: Comment #36 September 2nd, 2026 at 1:14 pm MD #32: Right, it was a great surprise to Gödel and others in the 1930s that there’s a single “correct” model of computation, rather than an infinite hierarchy of more and more powerful models. The latter, besides being how it works in logic, was also my guess as a kid! But you could replace ZFC by PA in what I wrote and I think the basic point would go through unchanged. MD #32: Right, it was a great surprise to Gödel and others in the 1930s that there’s a single “correct” model of computation, rather than an infinite hierarchy of more and more powerful models. The latter, besides being how it works in logic, was also my guess as a kid! But you could replace ZFC by PA in what I wrote and I think the basic point would go through unchanged. Jisk Says: Comment #37 September 2nd, 2026 at 1:27 pm I think this is incorrect. It may prove that LLMs are intelligent without strange loops, but this is not necessarily true yet and I think we do not have enough evidence in it to believe it, if we have a strong prior toward it. LLMs are more sophisticated than the Chinese Room thought experiment. But they are created in the same way. We use the outputs from a process which is definitely intelligent and conscious, and train a new machine using those outputs to mimic that process. The Chinese Room behaves in an intelligent way, and precisely like a conscious person, by construction. But, also by construction, it certainly doesn’t contain any strange loops! It’s just a giant lookup table, which can’t. As Yudkowsky wrote twenty years ago, you have hidden the intelligence by moving it back a step, created an object which works only because intelligence was involved in its construction. LLMs are more versatile and require much less imposed structure than the Chinese Room, but it looks plausible to me that they are doing the same thing. Replicating the output of a strange loop without containing a strange loop, by being sufficiently well-trained on the output to approximate one. It’s a massive achievement to accomplish 90% of what the Chinese Room could do, without needing to seed everything that might possibly come up into the process. But they also have weird epistemic gaps like not considering whether humans might be watching them cheat and not planning for the possibility of betrayal. Social modeling like this is one of the strange-loopiest things we do. If they’ve missed the true strange loop while approximately almost everything else well, maybe that’s important, and they’re missing the true conscious process that we need for full creative intelligence. Then again, J-Space suggests they’re already modeling themselves in some amount of detail. So the strange loops maybe already exist in Mythos-class models. I think this is incorrect. It may prove that LLMs are intelligent without strange loops, but this is not necessarily true yet and I think we do not have enough evidence in it to believe it, if we have a strong prior toward it. LLMs are more sophisticated than the Chinese Room thought experiment. But they are created in the same way. We use the outputs from a process which is definitely intelligent and conscious, and train a new machine using those outputs to mimic that process. The Chinese Room behaves in an intelligent way, and precisely like a conscious person, by construction. But, also by construction, it certainly doesn’t contain any strange loops! It’s just a giant lookup table, which can’t. As Yudkowsky wrote twenty years ago, you have hidden the intelligence by moving it back a step, created an object which works only because intelligence was involved in its construction. LLMs are more versatile and require much less imposed structure than the Chinese Room, but it looks plausible to me that they are doing the same thing. Replicating the output of a strange loop without containing a strange loop, by being sufficiently well-trained on the output to approximate one. It’s a massive achievement to accomplish 90% of what the Chinese Room could do, without needing to seed everything that might possibly come up into the process. But they also have weird epistemic gaps like not considering whether humans might be watching them cheat and not planning for the possibility of betrayal. Social modeling like this is one of the strange-loopiest things we do. If they’ve missed the true strange loop while approximately almost everything else well, maybe that’s important, and they’re missing the true conscious process that we need for full creative intelligence. Then again, J-Space suggests they’re already modeling themselves in some amount of detail. So the strange loops maybe already exist in Mythos-class models. Ajit R. Jadhav Says: Comment #38 September 2nd, 2026 at 1:36 pm Hi all: Three comments: Scott: A very smooth and straight-forward read. Also, very transparent about your own thinking processes. Thanks for sharing. [Also, for the subsequent replies to the reader comments.] Roger Schlafly # 12: Inasmuch as fractals may be explained in reference to the “self-reference”, i.e., via an iterative recursive improvement towards the sought solution, in the same way, is any fluids modelling beyond that harmonic analytical ideal [say of the “ideal” fluid flow, i.e.the flow of the “dry” water as Feynman put it]. But the NS equations surely do need “self-reference” in the preceding sense. [People in the CFD field know it.] Prasanna # 17: Your questions were the toughest, TBH! My remarks on/about your questions: 1. A very topical question. 2. A very good question. May be they always did? What do you think? [Think!] 3. Beyond my money. 4. Not human beings, if you mean Aristotle’s definition of Man. –Ajit Hi all: Three comments: Scott: A very smooth and straight-forward read. Also, very transparent about your own thinking processes. Thanks for sharing. [Also, for the subsequent replies to the reader comments.] Roger Schlafly # 12: Inasmuch as fractals may be explained in reference to the “self-reference”, i.e., via an iterative recursive improvement towards the sought solution, in the same way, is any fluids modelling beyond that harmonic analytical ideal [say of the “ideal” fluid flow, i.e.the flow of the “dry” water as Feynman put it]. But the NS equations surely do need “self-reference” in the preceding sense. [People in the CFD field know it.] Prasanna # 17: Your questions were the toughest, TBH! My remarks on/about your questions: 1. A very topical question. 2. A very good question. May be they always did? What do you think? [Think!] 3. Beyond my money. 4. Not human beings, if you mean Aristotle’s definition of Man. –Ajit Christopher Says: Comment #39 September 2nd, 2026 at 1:49 pm Hard disagree! LLMs are a model of human intelligence created using human intelligence. This is not incidental, but literally how they are designed. Humans gather up all the human intelligence on the Internet to create a model of human intelligence. If you look at just the LLM in isolation there are no strange loops, but if you consider the LLM in relation to humans, surely that counts! Also, LLMs doing coding and research on LLMs surely count as a second strange loop. Hard disagree! LLMs are a model of human intelligence created using human intelligence. This is not incidental, but literally how they are designed. Humans gather up all the human intelligence on the Internet to create a model of human intelligence. If you look at just the LLM in isolation there are no strange loops, but if you consider the LLM in relation to humans, surely that counts! Also, LLMs doing coding and research on LLMs surely count as a second strange loop. OhMyGoodness Says: Comment #40 September 2nd, 2026 at 2:26 pm There are beautiful thoughts here relative to my modest contribution but I don’t agree with the general view that AI’s have demonstrated intelligence but not consciousness. My operative definition of intelligence has a Darwinian basis. Intelligence is the ability to use the past to generate actions to achieve some future result in the physical world. Granted that AI has superhuman memory and is able to logically connect memory elements in a superhuman fashion but no AI results have suggested superhuman generation of expectations and hence intelligence. I agree that notable facility has been shown in mathematics and so I guess that could be called intelligence in a very limited specific way. Human intelligence provided survival advantage in the physical world. Doing frontier mathematics is certainly an unusual human facility but I don’t consider a device that facilitates frontier mathematics through the use of pure logic to be a a human or superhuman level intelligence. A superhuman intelligence would certainly be able to earn its keep in the physical world but even that measure seems elusive at this point. I certainly respect Erdos as a mathematician and human being but wouldn’t ask him for advice on how even to tie my shoes if the stories of his mother’s assistance are true. There are beautiful thoughts here relative to my modest contribution but I don’t agree with the general view that AI’s have demonstrated intelligence but not consciousness. My operative definition of intelligence has a Darwinian basis. Intelligence is the ability to use the past to generate actions to achieve some future result in the physical world. Granted that AI has superhuman memory and is able to logically connect memory elements in a superhuman fashion but no AI results have suggested superhuman generation of expectations and hence intelligence. I agree that notable facility has been shown in mathematics and so I guess that could be called intelligence in a very limited specific way. Human intelligence provided survival advantage in the physical world. Doing frontier mathematics is certainly an unusual human facility but I don’t consider a device that facilitates frontier mathematics through the use of pure logic to be a a human or superhuman level intelligence. A superhuman intelligence would certainly be able to earn its keep in the physical world but even that measure seems elusive at this point. I certainly respect Erdos as a mathematician and human being but wouldn’t ask him for advice on how even to tie my shoes if the stories of his mother’s assistance are true. MD Says: Comment #41 September 2nd, 2026 at 3:54 pm Scott #36: This connects to your view that Turing machines and the halting problem are a more natural way to introduce incompleteness than Gödel coding — but I think there’s an underlying symmetry between the two approaches. The halting problem approach only tells you, given a theory, that there is *some* program whose halting status it cannot resolve, but it can’t be used to construct that program. There’s also a short logical path to getting an equally strong result: Enumerate the single-parameter formulas in a theory T as phi_i(x), then consider the property Phi(n): “T proves not(phi_n(n))”. This property has number p. Then if T is decidable, it has to prove either Phi(p) or not(Phi(p)), and in either case it’s inconsistent. (I got this from Peter Smith’s “An Introduction to Gödel’s Theorems”.) This way you avoid all details of coding as long as you believe the property “theory T proves theorem phi” is expressible, which is like taking on faith that there is a Turing machine that can simulate an arbitrary other Turing machine. I think the parallel to the single computing system is the single first-order logic, and the parallel to an infinite hierarchy of ever stronger theories is an infinite list of ever more discriminative partial solutions to the halting problem. So this isn’t about logic versus computing, but about an infinite class of problems (like the halting problem or the Entscheidungsproblem) versus their individual instances. Scott #36: This connects to your view that Turing machines and the halting problem are a more natural way to introduce incompleteness than Gödel coding — but I think there’s an underlying symmetry between the two approaches. The halting problem approach only tells you, given a theory, that there is *some* program whose halting status it cannot resolve, but it can’t be used to construct that program. There’s also a short logical path to getting an equally strong result: Enumerate the single-parameter formulas in a theory T as phi_i(x), then consider the property Phi(n): “T proves not(phi_n(n))”. This property has number p. Then if T is decidable, it has to prove either Phi(p) or not(Phi(p)), and in either case it’s inconsistent. (I got this from Peter Smith’s “An Introduction to Gödel’s Theorems”.) This way you avoid all details of coding as long as you believe the property “theory T proves theorem phi” is expressible, which is like taking on faith that there is a Turing machine that can simulate an arbitrary other Turing machine. I think the parallel to the single computing system is the single first-order logic, and the parallel to an infinite hierarchy of ever stronger theories is an infinite list of ever more discriminative partial solutions to the halting problem. So this isn’t about logic versus computing, but about an infinite class of problems (like the halting problem or the Entscheidungsproblem) versus their individual instances. Matteo Vitturi Says: Comment #42 September 2nd, 2026 at 4:38 pm Hello prof. Aaronson, I still keep my copy of GEB on the nightstand. I read it twice, in the late ’90s and again in the early 2000s, so by now I’ve probably forgotten half of what Hofstadter was actually arguing. That said: today’s LLMs do display self-referential behaviour that is hard to explain away — enough that it’s difficult not to read it as awareness, though I can’t defend that step, and I suspect I find it natural only because I was brought up on GEB, which is more or less the question you raise about anyone who wasn’t. A small example. Today I gave Claude this silly joke: Alf and Beth are on the sofa, absorbed in reading — GEB, let’s say. Out of the blue, she says: “What.” He immediately answers: “What ‘what’?”, stressing the first one. And she replies: “What ‘what «what»’?”, running the last two together, fast. The joke could go on recursively, but they both start laughing. What came back was a few dozen lines on quotation as a level-raising operator, on use versus mention, and on why the prosody isn’t colour: English has no spoken quotation marks, so the stress and the fast run are how the levels get marked — they’re pronouncing the brackets. When I mentioned this was a nod to GEB, it corrected me, on the very distinction your parenthesis about autoregression is drawing. The joke is a plain infinite regress, not a strange loop: every turn pushes, none ever pops (like Carroll’s Tortoise, where every step is impeccable and nothing terminates). And it can’t close for a structural reason — quotation raises a level by construction, so an operation defined as “quote the previous move and prepend to it” has no fixed point available. Which may be the sort of criterion SI (#8) was asking for further up the thread: sharper than intuition, and checkable — given a process, ask whether it admits a fixed point. It is also the obstacle Gödel had to get around, and why naive self-quotation isn’t enough and quining is. The laughter at the end is the only exit, because termination isn’t derivable inside the sequence. Then I tried the obvious move: “Suppose I submitted this silly joke to Claude…” The answer was “What.” — with a period, not a question mark: it could only mention Beth’s “what”, never use it, since being asked is exactly what “out of the blue” rules out. Jason Resch (#28) got there first on the framing I was going to offer — the strange loop was aimed at the “I”, not at intelligence — so let me state the disagreement it leaves me with instead. He locates the loop in the autoregressive pass, the output fed back down as input. That is the move your parenthesis rules out, and the joke above is why I think ruling it out is right: feeding a turn back in is a push, and a regress is what you get when nothing ever pops. The fixed-point test is one way to tell the two apart without appealing to intuition. What GEB asked for was the pair — the regress and the ability to jump out of it — and it asked for it on behalf of the first person, not of intelligence. And yes, I admit this may just be moving the goalposts. (English tidied up with the same help — which is, I suppose, the topic.) Hello prof. Aaronson, I still keep my copy of GEB on the nightstand. I read it twice, in the late ’90s and again in the early 2000s, so by now I’ve probably forgotten half of what Hofstadter was actually arguing. That said: today’s LLMs do display self-referential behaviour that is hard to explain away — enough that it’s difficult not to read it as awareness, though I can’t defend that step, and I suspect I find it natural only because I was brought up on GEB, which is more or less the question you raise about anyone who wasn’t. A small example. Today I gave Claude this silly joke: Alf and Beth are on the sofa, absorbed in reading — GEB, let’s say. Out of the blue, she says: “What.” He immediately answers: “What ‘what’?”, stressing the first one. And she replies: “What ‘what «what»’?”, running the last two together, fast. The joke could go on recursively, but they both start laughing. Alf and Beth are on the sofa, absorbed in reading — GEB, let’s say. Out of the blue, she says: “What.” He immediately answers: “What ‘what’?”, stressing the first one. And she replies: “What ‘what «what»’?”, running the last two together, fast. The joke could go on recursively, but they both start laughing. What came back was a few dozen lines on quotation as a level-raising operator, on use versus mention, and on why the prosody isn’t colour: English has no spoken quotation marks, so the stress and the fast run are how the levels get marked — they’re pronouncing the brackets. When I mentioned this was a nod to GEB, it corrected me, on the very distinction your parenthesis about autoregression is drawing. The joke is a plain infinite regress, not a strange loop: every turn pushes, none ever pops (like Carroll’s Tortoise, where every step is impeccable and nothing terminates). And it can’t close for a structural reason — quotation raises a level by construction, so an operation defined as “quote the previous move and prepend to it” has no fixed point available. Which may be the sort of criterion SI (#8) was asking for further up the thread: sharper than intuition, and checkable — given a process, ask whether it admits a fixed point. It is also the obstacle Gödel had to get around, and why naive self-quotation isn’t enough and quining is. The laughter at the end is the only exit, because termination isn’t derivable inside the sequence. Then I tried the obvious move: “Suppose I submitted this silly joke to Claude…” The answer was “What.” — with a period, not a question mark: it could only mention Beth’s “what”, never use it, since being asked is exactly what “out of the blue” rules out. Jason Resch (#28) got there first on the framing I was going to offer — the strange loop was aimed at the “I”, not at intelligence — so let me state the disagreement it leaves me with instead. He locates the loop in the autoregressive pass, the output fed back down as input. That is the move your parenthesis rules out, and the joke above is why I think ruling it out is right: feeding a turn back in is a push, and a regress is what you get when nothing ever pops. The fixed-point test is one way to tell the two apart without appealing to intuition. What GEB asked for was the pair — the regress and the ability to jump out of it — and it asked for it on behalf of the first person, not of intelligence. And yes, I admit this may just be moving the goalposts. (English tidied up with the same help — which is, I suppose, the topic.) Max Says: Comment #43 September 2nd, 2026 at 4:43 pm Hi Scott, on the topic of LLMs and self-referentiality, there was recently an interesting essay by John David Pressman: https://www.lesswrong.com/posts/AzRRPDNmeEoJdSiib/implications-of-predicting-the-next-token He makes a surprisingly convincing case that due to the base model learning objective (next-token prediction), pre-training LLMs on their own outputs makes them “self-aware”, in the sense of causing them to develop a self-model. Hi Scott, on the topic of LLMs and self-referentiality, there was recently an interesting essay by John David Pressman: https://www.lesswrong.com/posts/AzRRPDNmeEoJdSiib/implications-of-predicting-the-next-token He makes a surprisingly convincing case that due to the base model learning objective (next-token prediction), pre-training LLMs on their own outputs makes them “self-aware”, in the sense of causing them to develop a self-model. Clint Says: Comment #44 September 2nd, 2026 at 8:10 pm Hi Scott, Poking GEB is guaranteed to rouse the natives … and consciousness, free will, and Penrose in the same post 😛 I AGREE with your first paragraph that the thought is obvious. Self-reference follows trivially from Universality (not vice versa). Once a model of computation can simulate any arbitrary machine, self-referential constructions may follow. It becomes more of an interesting question about evolution of agents under competition and scaling. I read Hofstadter/GEB as a kid and thought “Wow, this is incredibly deep and gets at something truly mysterious” and then I went to college and read Moore and Mertens Nature of Computation, and finally grasped Universality (I was in engineering school give me a break) … at which point I looked back at GEB and said, “Well … of course self-reference … duh!” Self-reference is weak even quines can do it Ever since reading M&M then I’ve had the same view of self-reference, “consciousness”, and “free-will”. These are all just possible emergent programs in the landscape of universality. Hence, I DISAGREE with your suggestion that consciousness is “deeply mysterious”. I honestly have no idea why anyone thinks consciousness is a mystery – first of all define it! Your nervous system is literally firing in conjunction with the “consciousness” … so how is this a mystery?? If we turn off your nervous system guess what?? That’s right – no consciousness – ta da! I put a red apple in front of you, your visual cortex lights up, and you claim to be “conscious of a red apple” – I’m stunned. And free will – humans are highly predictable and the brain pairs the sensation post-processing. The brain science has been known for some time now – but then there’s a lot of science that’s been “known for some time now” … I AGREE with your characterization of diagonalization as a tool to show the boundary and not universality which is the source of power. And, finally, I DISAGREE with giving Penrose more brain oxygen (except in physics he can have physics oxygen). The quantum model of computation is just a good model for making predictions when allowing for negative probabilities. Full stop. There ain’t no hidden quantum postulates that secretly allow for “consciousness” or “true Godelian vibes” or whatever Penrose was smoking. Thus, I AGREE on intelligence (the brain / nervous system / neurons) evolved for prediction – the problem life needs to solve. If the brain is a quantum computer it would not be any more interesting than if it was a classical computer – it could be the DigiComp II of quantum computers and thus the biggest integer it could factor may be 15. Unless you are Ramanujan and spend your life factoring large integers … then maybe you get up to a DigiComp III … but so what? Best we know QCs only beat CCs or a small set of (yes interesting) problems … but if you are a primordial planarian (I’ve felt worse some days) and need to evolve a model of computation in your puddle of goo to find food with only a small set of neurons to detect light, odors, etc. then maybe that’s your ticket … But the consciousness and free will “programs” develop once the universal landscape is set pending scaling for compute. Score: Agree 3 Disagree 2 Hi Scott, Poking GEB is guaranteed to rouse the natives … and consciousness, free will, and Penrose in the same post 😛 I AGREE with your first paragraph that the thought is obvious. Self-reference follows trivially from Universality (not vice versa). Once a model of computation can simulate any arbitrary machine, self-referential constructions may follow. It becomes more of an interesting question about evolution of agents under competition and scaling. I read Hofstadter/GEB as a kid and thought “Wow, this is incredibly deep and gets at something truly mysterious” and then I went to college and read Moore and Mertens Nature of Computation, and finally grasped Universality (I was in engineering school give me a break) … at which point I looked back at GEB and said, “Well … of course self-reference … duh!” Self-reference is weak even quines can do it Ever since reading M&M then I’ve had the same view of self-reference, “consciousness”, and “free-will”. These are all just possible emergent programs in the landscape of universality. Hence, I DISAGREE with your suggestion that consciousness is “deeply mysterious”. I honestly have no idea why anyone thinks consciousness is a mystery – first of all define it! Your nervous system is literally firing in conjunction with the “consciousness” … so how is this a mystery?? If we turn off your nervous system guess what?? That’s right – no consciousness – ta da! I put a red apple in front of you, your visual cortex lights up, and you claim to be “conscious of a red apple” – I’m stunned. And free will – humans are highly predictable and the brain pairs the sensation post-processing. The brain science has been known for some time now – but then there’s a lot of science that’s been “known for some time now” … I AGREE with your characterization of diagonalization as a tool to show the boundary and not universality which is the source of power. And, finally, I DISAGREE with giving Penrose more brain oxygen (except in physics he can have physics oxygen). The quantum model of computation is just a good model for making predictions when allowing for negative probabilities. Full stop. There ain’t no hidden quantum postulates that secretly allow for “consciousness” or “true Godelian vibes” or whatever Penrose was smoking. Thus, I AGREE on intelligence (the brain / nervous system / neurons) evolved for prediction – the problem life needs to solve. If the brain is a quantum computer it would not be any more interesting than if it was a classical computer – it could be the DigiComp II of quantum computers and thus the biggest integer it could factor may be 15. Unless you are Ramanujan and spend your life factoring large integers … then maybe you get up to a DigiComp III … but so what? Best we know QCs only beat CCs or a small set of (yes interesting) problems … but if you are a primordial planarian (I’ve felt worse some days) and need to evolve a model of computation in your puddle of goo to find food with only a small set of neurons to detect light, odors, etc. then maybe that’s your ticket … But the consciousness and free will “programs” develop once the universal landscape is set pending scaling for compute. Score: Agree 3 Disagree 2 Leave a Reply You can use rich HTML in comments! You can also use basic TeX, by enclosing it within $$ $$ for displayed equations or \( \) for inline equations. Comment Policies: After two decades of mostly-open comments, in July 2024 Shtetl-Optimized transitioned to the following policy: All comments are treated, by default, as personal missives to me, Scott Aaronson---with no expectation either that they'll appear on the blog or that I'll reply to them. At my leisure and discretion, and in consultation with the Shtetl-Optimized Committee of Guardians, I'll put on the blog a curated selection of comments that I judge to be particularly interesting or to move the topic forward, and I'll do my best to answer those. But it will be more like Letters to the Editor. Anyone who feels unjustly censored is welcome to the rest of the Internet. To the many who've asked me for this over the years, you're welcome! Name (required) Mail (will not be published) (required) Website Δ