The history of science — and of progress — is a series of benefits that are direct results of new tools. A large part of our current longevity is due to the invention of the microscope. This new way of seeing opened up the microscopic world, a teeming universe we had no idea about, and from that view quickly came germ theory, and soon after, new ways to avoid many common fatal diseases. Hundreds of other advances were also birthed by the microscope, including our understanding of DNA. Similarly, the telescope not only opened the heavens to our inspection, it had a direct role in helping us devise the laws of physics, which in turn permitted harnessing the atom, developing GPS, and making cheap computers. Oscilloscopes, volt meters, barometers, cyclotrons — these are more than measuring tools; they are portals that open up new territories to be explored.
We are on the cusp of inventing a new cyclotron: artificial intelligence. Of course AI will usher in new ways to do stuff. We can offload chores we don’t want to do, but the greatest power will be in accomplishing things we had never imagined doing before. That new superpower will gradually revamp our society as we learn how best to employ it.
But a secondary revolution will come from using AI as a microscope: we will use it to see our own minds. AI will be a cyclotron that lets us inspect the mysterious particles of cognition swirling in human brains — bits that are completely invisible to us now. With this cyclotron we will be able to dissect thinking, take intelligence apart to see its components. We will be able to run endless experiments on whole and partial minds, experiments we can’t and won’t run on ourselves.
The space of possible minds in the universe is vast. Using AI as our scope, we will begin to populate that space, constructing artificial minds for specific purposes — ones that do math proofs, others for everyday robots, others to write stories, another to manage the planet’s climate. With the ability to look into minds, we can begin to build a theory of mind. Centuries ago the microscope gave us germ theory and cell theory. The telescope gave us gravity and relativity. The new scope of AI will give us a theory of intelligence.
Despite the great strides scientists have made in producing AI, we have no theory of intelligence. We don’t know how intelligence works, in humans or in machines. We don’t know what it is, or why it produces smartness. What is the smallest possible thing that will produce intelligence? We don’t know. Is there a universal ingredient shared between humans and machines? We don’t know. What is the metric for intelligence — how would we even quantify it? We don’t know. If we want more of it, what’s the formula? We don’t know. Is intelligence one thing or several — one dimension or many? We don’t know that either. Until we can predict what AI will do, we have no theory, and if we have no theory, we can never predict what AI will do. This is a problem.
Our ignorance about intelligence is vast. Take energy: does intelligence require a lot of it, or just a little? The supercomputer inside our own skull runs on about 25 watts, which hints that the minimum energy needed for intelligence might be small — and that the giant, controversial AI compute centers we’re building now may turn out to be a temporary blip rather than a permanent feature.
In the 1800s, the field of artificial power — steam engines — was hobbled for decades by the lack of a theory of heat. Tinkering engineers spent years on inefficient machines, costly detours, before arriving at the insight that power comes from temperature differential, not temperature alone. There were still engineering problems to solve after that — inventing high-pressure equipment, for one — but the theory told them which hardware problems were worth solving.
Like the theory of heat, a theory of intelligence will require engineers and theorists working side by side. It will be a nerd endeavor that requires building things in order to understand them. It will need a diverse team — neurobiologists, physicists, mathematicians, computer scientists, philosophers, hardware experts — and I expect a lot of foolish ideas will be pursued in order to discover general principles of intelligence.. This is basic science at its purest. It will take patience.
The payoff is a much better sense of where to focus attention and resources. We would know in advance, not after a $100 million training run, whether a given approach is near-optimal, or where the cognitive waste is happening. A theory tells you where the limits are. For instance, it could answer this: are today’s LLMs 1 percent of the way to what’s physically possible with a GPU, or 90 percent? That would be extremely useful to know. And since we’re about to invent thousands of new kinds of minds, a good theory would let us reliably engineer the right configuration for a given task — the way thermodynamics informs the design of an engine.
There’s a good chance a theory of intelligence will turn out to be necessary for real alignment. We may need to understand what the internal representation of thinking should look like before we can engineer core values — not just rules — into an AI mind. Right now we’re doing this by unscientific trial and error. We are trying anything we think of to get AIs aligned with our tricky goals: be creative, but don’t do anything stupid or bad. An explicit formula of intelligence, with a sense of its tradeoffs and limits, would give us something to steer by.
But a theory of intelligence isn’t just for AI. We are thinking machines too, and our intelligence sometimes needs fixing. Modern medicine still classifies psychiatric and neurodegenerative conditions largely by symptom cluster (the DSM approach) rather than by which specific computational function has failed. Unlike brains, artificial systems can be opened, probed, disturbed, and modified to isolate function directly, at a level of access neuroscience has never had. I’d bet we learn more about our own brains from building a thousand AIs under a real theory of intelligence than we’ve learned from a century of neuroscience.
A theory of intelligence might also finally decouple intelligence from consciousness. Intelligence is likely a computational quantity; consciousness is a phenomenal one. Of course, there is a chance that we might discover there is no general theory of intelligence at all; that like life, intelligence is a squishy, messy, almost illusionary phenomenon that cannot be encapsulated into a mathematical formula. That would be unfortunate, but good to know sooner rather than later.
Right now a small group of scientists are trying to find a new field: the science of intelligence. Jacob Yates, a neuroscientist at UC Berkeley and one of the effort’s conveners, says there are already glimmers of where a theory might emerge — pointing to work connecting stochastic thermodynamics with variational inference, and to geometric tools for characterizing the promises of what can be learned from data. A theory might suggest that every unit of learning would need X amount of energy, and Y bits of data. It might propose diminishing returns on scale, or the theoretical limits on how smart anything can get.
Information theory transformed electronics, guiding and accelerating everything that followed. A theory of intelligence would do the same for work, research and science itself. However with or without a theory of intelligence, artificial intelligence is becoming our new cyclotron. It is a telescope that is opening up a new territory: the continent of minds. Once the most mysterious force in our lives – our minds, all minds – will then be revealed. Indeed, the most complex things in the known universe are now available for exploration and study. The entire realm of learning, smartness, and thinking will be near, accessible in the most practical way. In the long term the instrument of AI will probably exceed the importance of the microscope, telescope and cyclotron combined. We find it hard to see the real world without views of the ultra tiny and ultra large made possible by our tools. Future generations will find it hard to see the real world without views of all the possible minds operating upon it, each seeing the world in a slightly different way. A world without the tools of AIs will be unthinkable.
Hey Kevin, big fan of your work. I'd like to suggest that the theory of intelligence is quite a bit more advanced than this post claims. In particular, the Bayesian mechanics paradigm and core formalism -- built around a definition of agents as dynamical systems, with well-defined integrations into information and control theory -- have been introduced and hardened over the last 20 years by Karl Friston and others. Karl himself, a neuroscientist, originally focused on this as explicitly a paradigm for intelligence in living beings. The thermodynamic and other physical constraints have been established by Chris Fields; the biological link, by Michael Levin; etc.
Incidentally, this is also a paradigm for theories of life and "beingness" in general; one finds no bright-line distinction between intelligence in humans and in cells.
The paradigm does highlight that intelligence is a property of agents in an environment; disembodied intelligence makes no sense. The way this is resolved (as is needed to reason about LLMs and other AI models that promise to "have intelligence" as if it were a thing) is by thinking of an agent's internal state as performing amortized inference over its *possible* environments; the agent's intelligence in an actual environment is then determined in large part by the model's capability, how well it predicts the environment in the ways that matter - plus, as importantly, by its wiring to other subsystems and I/O to the world. This, then, provides a principled basis for the recent finding in LLM-land that "the harness is as important as the model".
It is a phenomenal read. Realising how close we feel, yet far away we actually are from this marvellous term "AGI" makes this era of the big guessing game on what Intelligence is and how it works even more ridiculous.