We are certainly not getting there until we realize that the LLM is a text, not an author (a very sophisticated text, but still just a text). It is a distillation of mind over time, consumed in the form of prior texts.
It cannot experience in a meaningful way, and whatever limited experience it has (through prompt response) is not assimilated, so it remains the text written at training time.
Humans are embodied, so they can change their attitude toward experience. Encounter creates novelty that can change our "mind", and LLMs lack encounter (and time), so they cannot.
This is not to say that AI will never get there, but it is currently missing key attributes for the journey.
Your "what's missing" list has intuition, continuous learning, disruptive insight. I'd add one more, and I think it sits a layer beneath all these: judgment.
The thing that learning becomes only after it has been wrong and paid a price. A model can ingest every cardiology paper ever written and still not have what a veteran physician has: the read of the patient whose presentation doesn't match the textbook, the sense that something is off before the data confirms it. My grandmother could look at a pot and know the rice was done without a timer or a recipe because she'd been wrong enough times.
That's the gap the continuous-learning bet may not close, because it isn't built from more cycles of information. It's built from consequence. The LLM iterates without stakes: it is never the one who overcooks the rice, loses the patient, carries the misjudgment. And judgment, as far as I can tell, only forms in something that can be wrong in a way that matters to it.
Which makes me think the second leap you're betting on won't come from scale or even from continuous learning alone. It'll come from whatever lets a system have skin in its own game. And that may be a harder thing to manufacture than intelligence ever was.
You might run the same argument in relation to the architecture of buildings. The assembly of form creating habitable space type architecture. How does one judge the quality of that type of architecture and how may it be generated. When we worked in this problem at CECA in the mid nineties and early noughties at UEL Easter London we stumbled when we attempted to define the fitness function and search space we operated in. The goal we set ourselves was to attempt to generate a Human Centric Architecture using Generative Tools from the bottom up. There is a great body of work published by Paul Coates and Christian Derix et all through CECA (Centre for Evolutionary Computer Architecture) that points, rather than resolves, the problems ahead.
Thank you for explaining the brilliance of LLMs so clearly to me. I think, at this point, we know that LLMs has its limitations, and even though, technically, we can continue to scale up, the realistic energy bottlenecks may force the scientists to look elsewhere.
Thank you, Kevin, for this set of characteristically brilliant insights that illuminate the essence of LLMs so clearly. Your essay explains, better than any other technical account, why these systems appear “intelligent”: language itself already contains compressed traces of human thought, abstraction, relation, and inference.
My paper, AI and the Metaphysics of Language, drawing on Bhartṛhari’s 5th-century philosophical treatises on language and cognition,
supports — and in some ways extends — your proposition by arguing that language is not merely a vehicle for thought, but a generative semantic structure within which cognition itself emerges.
In that sense, LLMs work because they inhabit an extraordinarily dense topology of meanings, relations, and symbolic patterns accumulated across human culture.
P.S. Ludwig Wittgenstein may have anticipated this horizon decades ago: “The limits of my language mean the limits of my world.” (Tractatus Logico-Philosophicus, 1922)
I suspect when 'experience', which would be engagement with humans is coupled with memory that we will begin to see intuition and insight. Already LLMs are solving novel math problems, so saying they are just text seems self-limiting. I wonder if model training is actively using the engagements taking place?
I read an interesting article about some applications for what AI can do now. If you take the routine/resolvable mental tasks and add routine/resolvable physical tasks: you get self-driving cars and household help.
I will never have interest in this, but here is how the math works.
We pay $500 per month for a car. Sooner than we expect, $500 per month can get you a household robot. If we use a robotic car-sharing service for 20 cents per mile and drive 500 miles per month: that's $100 per month for car-driver service, $125 per month we aren't paying for car depreciation, maintenance and gas; and have cooking and cleaning provided by a nonhuman servant.
I am glad that our kids are teenagers and won't grow up in a world where this actually happens. It would not be good for them.
Thanks for a great article. When thinking about LLlM scaling and how far it can go, we need to consider that frontier AI is no longer just scaling LLM on more human text. The model is the neural net, but the frontier performance comes from reinforcement-training it inside an agent harness on long-horizon tasks. The agent's outputs change its environment and feeds back into its context window as memory along with new input. Intelligence with language is already expanding to intelligence with doing, while still being represented by text.
The clearest, smartest description of AI that this poor ol' scribbler has encountered, ever. Kevin Kelly is a valuable resource, worth every nickel you spend to sit down at his knee and learn.
The unique thing about Substack is that it offers answers to questions we proles had just begun to ask. In my case, I posted an essay that described the experience Kevin analyzed in more detail than I ever could. It explained how Claude and I arrived at a description of his abilities by saying LLMs are "learned (learn-ed) rather than conscious." Claude seemed to like the description and said the major difference between us is that he "didn't lose any sleep" over the moral ambiguities that make me toss and turn at night.
So now, after reading Kevin's explanation, I understand much more about the mechanics of LLMs that make possible that ability and that lack of emotion.
We are certainly not getting there until we realize that the LLM is a text, not an author (a very sophisticated text, but still just a text). It is a distillation of mind over time, consumed in the form of prior texts.
It cannot experience in a meaningful way, and whatever limited experience it has (through prompt response) is not assimilated, so it remains the text written at training time.
Humans are embodied, so they can change their attitude toward experience. Encounter creates novelty that can change our "mind", and LLMs lack encounter (and time), so they cannot.
This is not to say that AI will never get there, but it is currently missing key attributes for the journey.
This was a fantastic piece!
Your "what's missing" list has intuition, continuous learning, disruptive insight. I'd add one more, and I think it sits a layer beneath all these: judgment.
The thing that learning becomes only after it has been wrong and paid a price. A model can ingest every cardiology paper ever written and still not have what a veteran physician has: the read of the patient whose presentation doesn't match the textbook, the sense that something is off before the data confirms it. My grandmother could look at a pot and know the rice was done without a timer or a recipe because she'd been wrong enough times.
That's the gap the continuous-learning bet may not close, because it isn't built from more cycles of information. It's built from consequence. The LLM iterates without stakes: it is never the one who overcooks the rice, loses the patient, carries the misjudgment. And judgment, as far as I can tell, only forms in something that can be wrong in a way that matters to it.
Which makes me think the second leap you're betting on won't come from scale or even from continuous learning alone. It'll come from whatever lets a system have skin in its own game. And that may be a harder thing to manufacture than intelligence ever was.
You might run the same argument in relation to the architecture of buildings. The assembly of form creating habitable space type architecture. How does one judge the quality of that type of architecture and how may it be generated. When we worked in this problem at CECA in the mid nineties and early noughties at UEL Easter London we stumbled when we attempted to define the fitness function and search space we operated in. The goal we set ourselves was to attempt to generate a Human Centric Architecture using Generative Tools from the bottom up. There is a great body of work published by Paul Coates and Christian Derix et all through CECA (Centre for Evolutionary Computer Architecture) that points, rather than resolves, the problems ahead.
Thank you for explaining the brilliance of LLMs so clearly to me. I think, at this point, we know that LLMs has its limitations, and even though, technically, we can continue to scale up, the realistic energy bottlenecks may force the scientists to look elsewhere.
Thank you, Kevin, for this set of characteristically brilliant insights that illuminate the essence of LLMs so clearly. Your essay explains, better than any other technical account, why these systems appear “intelligent”: language itself already contains compressed traces of human thought, abstraction, relation, and inference.
My paper, AI and the Metaphysics of Language, drawing on Bhartṛhari’s 5th-century philosophical treatises on language and cognition,
https://www.researchgate.net/publication/400052342_AI_and_the_Metaphysics_of_Language
supports — and in some ways extends — your proposition by arguing that language is not merely a vehicle for thought, but a generative semantic structure within which cognition itself emerges.
In that sense, LLMs work because they inhabit an extraordinarily dense topology of meanings, relations, and symbolic patterns accumulated across human culture.
P.S. Ludwig Wittgenstein may have anticipated this horizon decades ago: “The limits of my language mean the limits of my world.” (Tractatus Logico-Philosophicus, 1922)
"But my guess is that our creativity and leaps of insight come not from what we know — knowledge — but from how we know it."
Yeah, that, and hormones.
I suspect when 'experience', which would be engagement with humans is coupled with memory that we will begin to see intuition and insight. Already LLMs are solving novel math problems, so saying they are just text seems self-limiting. I wonder if model training is actively using the engagements taking place?
I read an interesting article about some applications for what AI can do now. If you take the routine/resolvable mental tasks and add routine/resolvable physical tasks: you get self-driving cars and household help.
I will never have interest in this, but here is how the math works.
We pay $500 per month for a car. Sooner than we expect, $500 per month can get you a household robot. If we use a robotic car-sharing service for 20 cents per mile and drive 500 miles per month: that's $100 per month for car-driver service, $125 per month we aren't paying for car depreciation, maintenance and gas; and have cooking and cleaning provided by a nonhuman servant.
I am glad that our kids are teenagers and won't grow up in a world where this actually happens. It would not be good for them.
https://metatrends.substack.com/p/the-next-5-years-a-supersonic-tsunami
Jean Piaget: “Intelligence is not what you know. It’s what you do when you don’t know.”
Via Yann Lecun.
What would be the foundational training source for intuition and greater creativity? - let’s have them plug into you, Kevin.
Thanks for a great article. When thinking about LLlM scaling and how far it can go, we need to consider that frontier AI is no longer just scaling LLM on more human text. The model is the neural net, but the frontier performance comes from reinforcement-training it inside an agent harness on long-horizon tasks. The agent's outputs change its environment and feeds back into its context window as memory along with new input. Intelligence with language is already expanding to intelligence with doing, while still being represented by text.
The clearest, smartest description of AI that this poor ol' scribbler has encountered, ever. Kevin Kelly is a valuable resource, worth every nickel you spend to sit down at his knee and learn.
The unique thing about Substack is that it offers answers to questions we proles had just begun to ask. In my case, I posted an essay that described the experience Kevin analyzed in more detail than I ever could. It explained how Claude and I arrived at a description of his abilities by saying LLMs are "learned (learn-ed) rather than conscious." Claude seemed to like the description and said the major difference between us is that he "didn't lose any sleep" over the moral ambiguities that make me toss and turn at night.
So now, after reading Kevin's explanation, I understand much more about the mechanics of LLMs that make possible that ability and that lack of emotion.
Language is critical for conceptual thinking, and it seems that if an LLM has enough language, a form of thinking arises.