Speculations on the Future of the Scientific Method
The following essay was published 20 years ago (January, 2006) on my blog The Technium. I edited the intro here, but the speculations are basically unchanged.
The invention of the scientific method is by far our greatest invention. From it pours millions of other inventions at a rapid pace. Without it, new things are only discovered by accident. With it, we continue to expand our knowledge with confidence and increased skill.
The scientific method is a process that combines different ideas, accumulated over many centuries of trial. Some of the most iconic ideas in the scientific method were only added recently, such as the randomized double blind experiment, and the placebo, both added within the last 80 years. The scientific method continues to evolve. We have missed its speed because we tend to associate science with academic journals and laboratories, which are mired in the past. But if we consider the scientific method as the generalized way in which we acquire information and structure knowledge, then we can see that this process is vast, rapid, modern, outpacing and underlying all other change. And this “structure of knowing” is due to be accelerated by ubiquitous AI.
There is a good chance the scientific method will evolve more in the next 80 years than in the past 80.
For instance, we are in the process of scanning all the 32 million books published by humans since the time of Sumerian clay tablets till now. Their true value will be unleashed as we hyperlink and cross-reference each idea in their pages – a technique long honored in research but never before practical on the scale of all-books. We have already digitized and linked all law in English, and half of the scholarly journals released in the last 25 years. This digitization enables machine translation to move knowledge from obscure languages to common ones. It enables text mining to discover patterns found in the library of libraries that cannot be seen book by book. These are but two small points in the transformation of information technology. We see daily accelerations in bandwidth, storage and search – each step hyped by glossy magazines and web blogs which marks them as amusements and diversions, which they are not. The entire frontier of computers, hyperlinks, wikis, search indexes, RFID tags, wi-fi, simulations, and the rest of the techno goodie bag are in fact reshaping the nature of science. They are tools of knowledge. First these innovations change what we know, and then they change how we know. Then they change how we change.
Not only will science continue to surprise us with what it discovers and creates, it will continue to modify itself so that it surprises us by new methods. At the core of science’s self-modification is technology. New tools enable new structures of knowledge and new ways of discovery. The scientific method hundreds of years from now will differ from today’s understanding as we add new ways of processing and testing information. As in biological evolution, new organizations are layered upon the old without displacement of the old. The present scientific methods are not jettisoned; they are subsumed by new levels of knowing stuff.
What are some technological changes that might enable us to discover, test, prove and know things in the next 80 years? Some of these technologies will alter the scientific method directly but others will arrive for other purposes and then will be found to change how we come to know things.
Based on my own active imagination, I offer the following as possible near-term advances in the evolution of the scientific method:
****
Compiled Negative Results – Negative results are saved, shared, compiled and analyzed, instead of being dumped. Positive results may increase their credibility when linked to negative results. We already have hints of this in the recent decision of biochemical journals to require investigators to register early phase 1 clinical trials. Usually phase 1 trials of a drug end in failure and their negative results are not reported. As a public health measure, these negative results should be shared, so journals have pledged not to publish the findings of phase 3 trials if their phase 1 results had not been reported, whether negative or not.
Triple Blind Experiments – All participants are blind to the fact that they are involved in an experiment during measurement. While ordinary life continues, massive amounts of data are drawn and archived. From this multitude of measurements, controls and variables are identified and “isolated” afterwards. For instance, the vital signs and lifestyle metrics of a hundred thousand people might be recorded non-invasively for 20-years, and then later analysis could find certain variables (smoking habits, heart conditions) that would permit the entire 20 years to be viewed as an experiment – one that no one knew was even going on.
Combinatorial Sweep Exploration – Much of the unknown can be explored by systematically creating random varieties of it at a large scale. You can explore a certain type of ceramic by creating all possible types of ceramic, and then testing them. You can explore certain realms of proteins by generating all possible variations of that type of protein and then seeing if they bind. You can discover new algorithms by automatically programming all possible programs and then running them. Indeed all possible Xs of almost any sort can be summoned and examined as a way to study X. The parameters of this “library” of possibilities become the experiment. With sufficient computational power, together with a pool of proper primitive parts, vast territories unknown to science can be probed.
Evolutionary Search – A combinatorial exploration can be taken even further. If new libraries of variations can be derived from the best of a previous generation of good results, it is possible to evolve solutions. The best results are mutated and bred toward better results. The best testing protein is mutated randomly in thousands of way, and the best of that bunch kept and mutated further, until a lineage of proteins, each one more suited to the task than its ancestors, finally leads to one that works perfectly. This method can be applied to computer programs and even hypotheses.
Multiple Hypothesis Matrix – Instead of proposing a single hypothesis, a matrix of hypothesis scenarios are proposed and managed. Many of these hypotheses may be algorithmically generated. But they are entertained simultaneously. An experiment travels through the matrix of multiple hypotheses, and more than one thesis is permitted to stand with the results. The multiple thesis are passed onto the next experiment.
Theory-less Pattern Augmentation – As we enter a world of massive abundant data, we can apply AI software to aid in detecting patterns without the need to have a theory first. Curve-fitting software which seeks out a pattern in statistical information is a precursor. But in large bodies of information with many variables, AI algorithms can discover emergent patterns, and from these patterns hypothesis can be generated. These already exist in specialized niches of knowledge (such particle smashing) but more general rules and engines will enable pattern seeking tools to become part of all data treatment.
Adaptive Real Time Experiments – Results evaluated, and large-scale experiments modified in real time. What we have now is primarily batch-mode science. Traditionally, the experiment starts, the results collected, and conclusions reached. Then the next experiment is designed in response, and launched. In adaptive experiments, the analysis happens in parallel with collection, and the intent and design of the test is shifted on the fly. Some medical tests are already stopped or re-evaluated on the basis of early findings; this method would extend that method to other realms. Proper methods would be needed to keep the adaptive experiment objective.
AI Proofs – AI to check the logic of the experiment. As science experiments become ever more sophisticated and complicated, they become ever more difficult to judge. Artificial expert systems will at first evaluate the scientific logic of a paper to ensure the architecture of the argument is valid, and that it publishes the required types of data. This will augment the opinions of editors and peer-reviewers, but over time as the protocols for an AI check became standard, AI can score many papers for certain consistencies.
Wiki-Science – Experiments involving thousands of investigators collaborating on a “paper.” The paper is ongoing, and never finished. It is really a trail of edits and experiments posted in real time to an evolving “document.” Contributions are not assigned. The average number of authors per paper continues to rise. With massive collaborations, the numbers will boom. Tools for tracking credit and contributions will be vital.
Defined Benefit Funding – The use of prize money for particular scientific achievements. A goal is defined, funding secured for the first to reach it, and the contest opened to all. This method can also be combined with prediction markets, which wager on possible winners, and can liberate further funds for development.
Zillionics – Ubiquitous 24/7 sensors in bodies and in the environment can transform medical and environmental sciences. Unrelenting rivers of sensory data will flow day and night from zillions of sources. The exploding number of new, cheap, wireless, and novel sensing tools will require new types of programs to distill, index and archive this ocean of data, as well as to find meaningful signals in it. The field of “zillionics” — dealing with zillions of data flows — will be essential in health, natural sciences, and astronomy.
Deep Simulations – As our knowledge of complex systems advances, we can construct more complex simulations of them. Both the success and failures of these simulations will help us to acquire more knowledge of the systems. Developing a robust simulation will become a fundamental part of science in every field. Indeed the science of making viable simulations will become its own specialty, with a set of best practices, and an emerging theory of simulations. And just as we now expect a hypothesis to be subjected to the discipline of being stated in mathematical equations, in the future we will expect all hypotheses to be exercised in a simulation.
Hyper-analysis Mapping – Just as meta-analysis gathered diverse experiments on one subject and integrated their (sometimes contradictory) results into a large meta-view, hyper-analysis creates an extremely large-scale view by pulling together meta-analysis. The cross-links of references, assumptions, evidence and results are unraveled by computation, and then reviewed at a larger scale. Hyper-mapping tallies not only what is known in a particular wide field, but also emphasizes unknowns and contradictions. It is used to spotlight ‘white spaces’ where additional research would be most productive.
Return of the Subjective – Science came into its own when it managed to refuse the subjective and embrace the objective. The repeatability of an experiment by another, perhaps less enthusiastic, observer was instrumental in keeping science rational. But as science plunges into the outer limits of scale – at the largest and smallest ends – and confronts the weirdness of the fundamental principles of matter/energy/information, it may not be able to ignore the role of the observer. Existence seems to be a paradox of self-causality, and any science exploring the origins of existence will eventually have to embrace the subjective, without becoming irrational. The tools for managing paradoxes are still undeveloped.
***********
These speculations of mine are likely to miss the mark. I would easily bet that the actual new methods of science present in 40 years from now will be ones no one has thought of today. But I am extremely certain there will be further advances and additions, to such an extent that today’s essential scientific method will seem primitive and crude.
That's an amazing list from 20 years ago.
In a recent podcast from Dwarkesh Patel, Terence Tao talked briefly about how AI is turning the scientific method on its head, whereby the data comes first (data collection from experiments or other sources), and hypotheses come second, generated by AI as it spins through the collected data.
I think this fits into your category "Theory-less Pattern Augmentation".
What Kevin Kelly Can't See
Kevin Kelly recently republished his essay speculating on the future of the scientific method. It's a smart list. Compiled negative results. Triple blind experiments. AI proofs. Evolutionary search. Zillionics. Twenty years on, a surprising number of his predictions have arrived or are arriving.
But Kelly wrote a post about the scientific method — lowercase s, lowercase m — and never touched Science. Capital S. The institution. The living thing. His entire essay is about how we might upgrade the engine. He never asks who's driving, where it's going, or who it's running over on the way.
That's the blind spot. And it's not just Kelly's. It's the signature blind spot of the entire Wired/Long Now/Santa Fe worldview: an extraordinary capacity to see and predict tools and an extraordinary inability to see power.
Here's what I mean.
Every one of Kelly's predictions — better data, smarter algorithms, adaptive experiments, wiki-science — improves the method. The technique. The procedure by which we extract reliable knowledge from reality. Fine. But Science is not a method. Science is a living institution. It recruits attention. It enforces boundaries. It accumulates memory across generations. It reproduces itself through training, funding, publication, and peer review. It persists beyond any individual. And like every living institution, it depends on resources it does not control: human cognition, public trust, physical infrastructure, political will, and — above all — money.
The method is how Science thinks. The institution is how Science eats. And what Science eats determines what Science thinks about.
This is the half Kelly left out. Not how we improve the method — but how we govern the institution. How we get Science to serve People instead of Markets.
Because right now, the direction of scientific inquiry is overwhelmingly set by the institutions that fund it. Pharmaceutical companies fund drug research that serves quarterly earnings. Defense departments fund physics that serves weapons programs. Tech platforms fund AI research that serves engagement metrics. The method inside each of these programs might be rigorous. The experiments might be well-designed. The data might be clean. And the direction — what questions get asked, what problems get funded, what knowledge gets produced — is determined by the survival needs of the institutions writing the checks.
Kelly's upgrades make the engine more powerful. They do nothing to change where the engine is pointed.
Kuhn saw the internal version of this problem. Normal science is the institution defending its own boundaries — enforcing protocols, training participants to see what the paradigm permits, suppressing anomalies that threaten the structure. But Kuhn was describing Science's immune system. The governance problem is different and worse: it's not that Science resists internal disruption. It's that Science has been captured by external institutions whose survival needs have nothing to do with what humanity actually needs to know.
The replication crisis isn't a methodological failure. It's a governance failure. Scientists aren't faking results because the method is bad. They're cutting corners because the funding structure rewards publication volume over reliability, because career incentives select for novelty over replication, because the institutions Science depends on for survival have optimized it for output rather than integrity. You can triple-blind every experiment on earth and it won't matter if the questions being asked are chosen by the wrong people for the wrong reasons.
So here's the question Kelly should have asked: what would it look like to evolve not just the scientific method, but the governance of Science itself?
What if the direction of scientific inquiry were set not by the institutions that profit from the answers, but by the people who have to live with the consequences? What if funding structures were designed to protect feedback integrity — the capacity of Science to tell us things we don't want to hear — rather than to maximize commercially viable output? What if we treated Science the way we treat other critical public infrastructure: as something too important to be governed primarily by market logic?
This isn't a utopian fantasy. It's an institutional design problem. We've solved versions of it before — central banks, public utilities, judicial independence — by creating structures that are publicly funded but operationally insulated from the short-term incentives of the institutions around them. We know how to build firewalls between funding and direction. We just haven't applied that knowledge to Science, because the Kellys of the world keep us dazzled by the tools while the governance rots underneath.
Kelly closes by saying today's scientific method will seem "primitive and crude" in forty years. He's probably right. But not because we'll have better algorithms. Because we'll finally understand that the method was never the thing that needed saving. The method is fine. It's the institution that's sick. And the cure isn't a better microscope. It's answering a question that Kelly, for all his speculative imagination, never thought to ask:
Who should Science belong to?