We are certainly not getting there until we realize that the LLM is a text, not an author (a very sophisticated text, but still just a text). It is a distillation of mind over time, consumed in the form of prior texts.
It cannot experience in a meaningful way, and whatever limited experience it has (through prompt response) is not assimilated, so it remains the text written at training time.
Humans are embodied, so they can change their attitude toward experience. Encounter creates novelty that can change our "mind", and LLMs lack encounter (and time), so they cannot.
This is not to say that AI will never get there, but it is currently missing key attributes for the journey.
??? LLMs certainly can encounter more material (and change their "minds") by training on new material. And why can it not train through prompt response?
Also, does it matter whether LLMs can "experience" when they can already making amazing discoveries (in math, science, etc.) simply by manipulating texts?
I certainly understand the desire to privilege our own human meat-based electrical and chemical computer (being human myself), but there's little reason to believe that our little human meat-based electrical and chemical computer will always stand at the apex.
LLMs can be retrained, but they do not change at inference time, and there are no good models to achieve this.
As such, their weightings are exactly the same each prompt, meaning they are not affected by the encounter.
Researchers are trying to figure out how to do this, but one problem they are going to find is how to incorporate 300million different encounters in realtime (isn’t going to happen soon).
You should understand that there is no training that happens after training, except building a new model.
Answering the encountering problem would take more than this comment, but suffice to say every living thing participates in alteration through encounter (that is how life works). If you are interested, read “Mind in Life” by Evan Thompson. It is amazing.
Obviously, LLMs are incredible, and they wildly outstrip our abilities in many domains. But they are not conscious, and they should not be considered human-like intelligences running on a different substrate. They are currently a tool, and one that has material biases that they cannot correct (because they cannot learn the way living things learn).
I am not arguing for humans at the apex. I am just pointing out that right now the LLM is a tool (specially a “text”) not an intelligent agent. Eventually, it could become more, and it could eliminate life on the planet.
But what’s currently built is not that.
Here is a prompt that will give more information:
“Why can’t current GPTs change their model weights based on prompts. Why is inference totally separate from training?”
Your "what's missing" list has intuition, continuous learning, disruptive insight. I'd add one more, and I think it sits a layer beneath all these: judgment.
The thing that learning becomes only after it has been wrong and paid a price. A model can ingest every cardiology paper ever written and still not have what a veteran physician has: the read of the patient whose presentation doesn't match the textbook, the sense that something is off before the data confirms it. My grandmother could look at a pot and know the rice was done without a timer or a recipe because she'd been wrong enough times.
That's the gap the continuous-learning bet may not close, because it isn't built from more cycles of information. It's built from consequence. The LLM iterates without stakes: it is never the one who overcooks the rice, loses the patient, carries the misjudgment. And judgment, as far as I can tell, only forms in something that can be wrong in a way that matters to it.
Which makes me think the second leap you're betting on won't come from scale or even from continuous learning alone. It'll come from whatever lets a system have skin in its own game. And that may be a harder thing to manufacture than intelligence ever was.
This is quite an interesting thesis. Structurally,our brains are quite similar to those of other mammals who display quite limited intelligence. Yet almost anything with a brain (and a few organisms without) are able to learn. The difference is humans are unique in having an expansive language.
Consider human babies as well- they learn very rapidly yet have limited intelligence as far as abstract thinking. Our brains universally have “infantile amnesia,” although there is no consensus on the cause. Perhaps LLMs have provided some empirical clues.
Your two bets — scale the net, or redesign it — both keep intelligence inside one mind. But the leap you say will be "equally unexpected" might be one you already named (if I am remembering it correctly 😬): the orchestration of intelligences from The Inevitable.
The recent "Reasoning Models Generate Societies of Thought" paper (recent Long Now speaker Blaise Agüera y Arcas among the authors) argues a single reasoning model already behaves like a small society — perspectives in conflict and reconciliation, not just an extended chain. If that holds, the second leap isn't a new substrate. It's organization. The plurality was hiding inside the thing we kept calling singular.
The part I'd press is "how we know, not what we know." You locate the missing continuous learning in the model. I keep finding it between a person and a well-instructed one (Claude project spaces with well defined instructions and context for specific spaces for thinking and collaboration) — a year-deep, curated spaces that have changed how I notice and think more than any output it's produced. The learning compounded in the relationship, not the engine. Maybe that's where to watch for the next surprise.
Thank you, Kevin, for this set of characteristically brilliant insights that illuminate the essence of LLMs so clearly. Your essay explains, better than any other technical account, why these systems appear “intelligent”: language itself already contains compressed traces of human thought, abstraction, relation, and inference.
My paper, AI and the Metaphysics of Language, drawing on Bhartṛhari’s 5th-century philosophical treatises on language and cognition,
supports — and in some ways extends — your proposition by arguing that language is not merely a vehicle for thought, but a generative semantic structure within which cognition itself emerges.
In that sense, LLMs work because they inhabit an extraordinarily dense topology of meanings, relations, and symbolic patterns accumulated across human culture.
P.S. Ludwig Wittgenstein may have anticipated this horizon decades ago: “The limits of my language mean the limits of my world.” (Tractatus Logico-Philosophicus, 1922)
I wonder if "aha" moments are entirely the product of continuous learning within an individual brain, or whether some emerge from the noosphere. Do you think breakthroughs are purely the result of iterative neural processes, or is it possible that we're tapping into patterns that exist beyond any one individual? Or is that taking the idea too far?
Indic interpretations (Upanishads) suggest that Consciousness itself may be connected in ways science hasn't established, although such ideas remain speculative and aren't supported by mainstream scientific evidence.
I think this is close, but I would make one distinction sharper: intelligence does not originate in language alone. Language is a late, compressed layer of a much older adaptive process.
Intelligence emerges from the loop of perception, action, error, correction, memory, and goal-directedness in a world that resists us. Human language contains the traces of that process: embodiment, perception, social practice, failure, repair, experiment, responsibility, and memory compressed into symbolic form. That is why LLMs are so powerful. They can extract and recombine these traces at scale.
But extracting the sediment of intelligence is not the same as possessing the full generative process that produced it. So perhaps the surprising thing is not that language creates intelligence, but that language preserves enough of the structure of human thinking for statistical systems to simulate parts of it. What remains missing is not merely more text, but continuity of learning, encounter with the world, and an evaluative state that can be changed by its own history.
I too believe language is essential for intelligence, the ability to symbolic expression and sharing of ideas.
LLM, with the harness/agentic part, are capable of continual improvement, and when that is applied to itself it's not clear the real limit it will hit. What it lacks is a more complete real world view, and that is something humans are good at. Anyway, each of us maintains an unique world view inside our skull that somehow allows us to function and collaborate and improve. I see no reason for that to be different with AI systems. Expect in near future that aspects like: sovereignity, identity/liability, human in the loop, non-repudiation... Will be sorted out at region and worldwide level
You might run the same argument in relation to the architecture of buildings. The assembly of form creating habitable space type architecture. How does one judge the quality of that type of architecture and how may it be generated. When we worked in this problem at CECA in the mid nineties and early noughties at UEL Easter London we stumbled when we attempted to define the fitness function and search space we operated in. The goal we set ourselves was to attempt to generate a Human Centric Architecture using Generative Tools from the bottom up. There is a great body of work published by Paul Coates and Christian Derix et all through CECA (Centre for Evolutionary Computer Architecture) that points, rather than resolves, the problems ahead.
Thank you for explaining the brilliance of LLMs so clearly to me. I think, at this point, we know that LLMs has its limitations, and even though, technically, we can continue to scale up, the realistic energy bottlenecks may force the scientists to look elsewhere.
We are certainly not getting there until we realize that the LLM is a text, not an author (a very sophisticated text, but still just a text). It is a distillation of mind over time, consumed in the form of prior texts.
It cannot experience in a meaningful way, and whatever limited experience it has (through prompt response) is not assimilated, so it remains the text written at training time.
Humans are embodied, so they can change their attitude toward experience. Encounter creates novelty that can change our "mind", and LLMs lack encounter (and time), so they cannot.
This is not to say that AI will never get there, but it is currently missing key attributes for the journey.
??? LLMs certainly can encounter more material (and change their "minds") by training on new material. And why can it not train through prompt response?
Also, does it matter whether LLMs can "experience" when they can already making amazing discoveries (in math, science, etc.) simply by manipulating texts?
I certainly understand the desire to privilege our own human meat-based electrical and chemical computer (being human myself), but there's little reason to believe that our little human meat-based electrical and chemical computer will always stand at the apex.
LLMs can be retrained, but they do not change at inference time, and there are no good models to achieve this.
As such, their weightings are exactly the same each prompt, meaning they are not affected by the encounter.
Researchers are trying to figure out how to do this, but one problem they are going to find is how to incorporate 300million different encounters in realtime (isn’t going to happen soon).
You should understand that there is no training that happens after training, except building a new model.
Answering the encountering problem would take more than this comment, but suffice to say every living thing participates in alteration through encounter (that is how life works). If you are interested, read “Mind in Life” by Evan Thompson. It is amazing.
Obviously, LLMs are incredible, and they wildly outstrip our abilities in many domains. But they are not conscious, and they should not be considered human-like intelligences running on a different substrate. They are currently a tool, and one that has material biases that they cannot correct (because they cannot learn the way living things learn).
I am not arguing for humans at the apex. I am just pointing out that right now the LLM is a tool (specially a “text”) not an intelligent agent. Eventually, it could become more, and it could eliminate life on the planet.
But what’s currently built is not that.
Here is a prompt that will give more information:
“Why can’t current GPTs change their model weights based on prompts. Why is inference totally separate from training?”
Jean Piaget: “Intelligence is not what you know. It’s what you do when you don’t know.”
Via Yann Lecun.
Your "what's missing" list has intuition, continuous learning, disruptive insight. I'd add one more, and I think it sits a layer beneath all these: judgment.
The thing that learning becomes only after it has been wrong and paid a price. A model can ingest every cardiology paper ever written and still not have what a veteran physician has: the read of the patient whose presentation doesn't match the textbook, the sense that something is off before the data confirms it. My grandmother could look at a pot and know the rice was done without a timer or a recipe because she'd been wrong enough times.
That's the gap the continuous-learning bet may not close, because it isn't built from more cycles of information. It's built from consequence. The LLM iterates without stakes: it is never the one who overcooks the rice, loses the patient, carries the misjudgment. And judgment, as far as I can tell, only forms in something that can be wrong in a way that matters to it.
Which makes me think the second leap you're betting on won't come from scale or even from continuous learning alone. It'll come from whatever lets a system have skin in its own game. And that may be a harder thing to manufacture than intelligence ever was.
This was a fantastic piece!
This is quite an interesting thesis. Structurally,our brains are quite similar to those of other mammals who display quite limited intelligence. Yet almost anything with a brain (and a few organisms without) are able to learn. The difference is humans are unique in having an expansive language.
Consider human babies as well- they learn very rapidly yet have limited intelligence as far as abstract thinking. Our brains universally have “infantile amnesia,” although there is no consensus on the cause. Perhaps LLMs have provided some empirical clues.
Your two bets — scale the net, or redesign it — both keep intelligence inside one mind. But the leap you say will be "equally unexpected" might be one you already named (if I am remembering it correctly 😬): the orchestration of intelligences from The Inevitable.
The recent "Reasoning Models Generate Societies of Thought" paper (recent Long Now speaker Blaise Agüera y Arcas among the authors) argues a single reasoning model already behaves like a small society — perspectives in conflict and reconciliation, not just an extended chain. If that holds, the second leap isn't a new substrate. It's organization. The plurality was hiding inside the thing we kept calling singular.
The part I'd press is "how we know, not what we know." You locate the missing continuous learning in the model. I keep finding it between a person and a well-instructed one (Claude project spaces with well defined instructions and context for specific spaces for thinking and collaboration) — a year-deep, curated spaces that have changed how I notice and think more than any output it's produced. The learning compounded in the relationship, not the engine. Maybe that's where to watch for the next surprise.
Thank you, Kevin, for this set of characteristically brilliant insights that illuminate the essence of LLMs so clearly. Your essay explains, better than any other technical account, why these systems appear “intelligent”: language itself already contains compressed traces of human thought, abstraction, relation, and inference.
My paper, AI and the Metaphysics of Language, drawing on Bhartṛhari’s 5th-century philosophical treatises on language and cognition,
https://www.researchgate.net/publication/400052342_AI_and_the_Metaphysics_of_Language
supports — and in some ways extends — your proposition by arguing that language is not merely a vehicle for thought, but a generative semantic structure within which cognition itself emerges.
In that sense, LLMs work because they inhabit an extraordinarily dense topology of meanings, relations, and symbolic patterns accumulated across human culture.
P.S. Ludwig Wittgenstein may have anticipated this horizon decades ago: “The limits of my language mean the limits of my world.” (Tractatus Logico-Philosophicus, 1922)
Language is critical for conceptual thinking, and it seems that if an LLM has enough language, a form of thinking arises.
I wonder if "aha" moments are entirely the product of continuous learning within an individual brain, or whether some emerge from the noosphere. Do you think breakthroughs are purely the result of iterative neural processes, or is it possible that we're tapping into patterns that exist beyond any one individual? Or is that taking the idea too far?
Indic interpretations (Upanishads) suggest that Consciousness itself may be connected in ways science hasn't established, although such ideas remain speculative and aren't supported by mainstream scientific evidence.
I think this is close, but I would make one distinction sharper: intelligence does not originate in language alone. Language is a late, compressed layer of a much older adaptive process.
Intelligence emerges from the loop of perception, action, error, correction, memory, and goal-directedness in a world that resists us. Human language contains the traces of that process: embodiment, perception, social practice, failure, repair, experiment, responsibility, and memory compressed into symbolic form. That is why LLMs are so powerful. They can extract and recombine these traces at scale.
But extracting the sediment of intelligence is not the same as possessing the full generative process that produced it. So perhaps the surprising thing is not that language creates intelligence, but that language preserves enough of the structure of human thinking for statistical systems to simulate parts of it. What remains missing is not merely more text, but continuity of learning, encounter with the world, and an evaluative state that can be changed by its own history.
I too believe language is essential for intelligence, the ability to symbolic expression and sharing of ideas.
LLM, with the harness/agentic part, are capable of continual improvement, and when that is applied to itself it's not clear the real limit it will hit. What it lacks is a more complete real world view, and that is something humans are good at. Anyway, each of us maintains an unique world view inside our skull that somehow allows us to function and collaborate and improve. I see no reason for that to be different with AI systems. Expect in near future that aspects like: sovereignity, identity/liability, human in the loop, non-repudiation... Will be sorted out at region and worldwide level
See ddmfca@substack “from Ochre to Algorithm “
Sapir Whorf hypothesis relevant here?
You might run the same argument in relation to the architecture of buildings. The assembly of form creating habitable space type architecture. How does one judge the quality of that type of architecture and how may it be generated. When we worked in this problem at CECA in the mid nineties and early noughties at UEL Easter London we stumbled when we attempted to define the fitness function and search space we operated in. The goal we set ourselves was to attempt to generate a Human Centric Architecture using Generative Tools from the bottom up. There is a great body of work published by Paul Coates and Christian Derix et all through CECA (Centre for Evolutionary Computer Architecture) that points, rather than resolves, the problems ahead.
Thank you for explaining the brilliance of LLMs so clearly to me. I think, at this point, we know that LLMs has its limitations, and even though, technically, we can continue to scale up, the realistic energy bottlenecks may force the scientists to look elsewhere.
"But my guess is that our creativity and leaps of insight come not from what we know — knowledge — but from how we know it."
Yeah, that, and hormones.