What if the weakest assumption may be equating coherence with truth?
History is full of highly coherent systems that later proved inaccurate:
→ geocentrism
→ racial pseudoscience
→ certain political ideologies
→ economic dogmas
Many beliefs become coherent simply because enough people repeat them.
Which raises a deeper question.
If AI learns from consensus-weighted coherence, how different is that from humans?
Much of what we call identity, reality, and truth is also inherited through conditioning, culture, repetition, and social reinforcement.
In that sense, the most provocative question may not be:
"Do AIs want to be honest?"
But rather:
"How much of what humans call truth, identity, and reality emerges from the coherence structures we were trained inside?"
That question led us to write a recent piece exploring how human identity itself can become artificially generated through adaptation, long before conscious authorship becomes possible.
Which raises an even deeper set of questions:
What if humans want to be honest, but can't?
And perhaps even more challenging:
Can humans tolerate the truth once it threatens the identity they were trained to become?
Interesting questions and thoughts here, Yoshi! Social coherence is one thing while scientific coherence is another; way back in the day, I think they used to be the same thing (or at least close/overlapping). But the scientific method is something that we discovered/invented and have slowly updated, like a tool that we invented and continually improved upon ever since. "Science," (AKA, the process of using the scientific method to acquire and test information to build a system of knowledge) has both been differentiating itself (from social coherence) and increasing its rigor/robustness as time goes on. As that has happened (and continues to happen) things like geocentrism, racial pseudoscience, certain political ideologies and economic dogmas become more clearly incoherent (scientifically speaking) even if they are socially coherent among a given population. I made the analogy of a crossword puzzle in my other comment on this (KK's) post. I don't know a lot about how LLMs work but I would hope that that they are able to differentiate social coherence from scientific coherence.
I enjoyed your question: "How much of what humans call truth, identity, and reality emerges from the coherence structures we were trained inside?" but it's not really the same question as what this article is getting at (at least in my mind). Though I'll check out your linked post about that since it's a fascinating (and adjacent) idea too!
Lee, I agree that we're exploring a slightly different question.
Kevin's piece seems primarily concerned with whether AI develops an attractor toward truth.
What Identity Agility is asking is something adjacent:
If humans are conditioned by family, culture, institutions, rewards, attachment systems, and narratives, why assume AI conditioning is fundamentally different?
Humans are trained. AI is trained.
Humans can mistake adaptation for truth. AI may also mistake adaptation for truth.
So the deeper question for us becomes:
Can any intelligence fully escape its training?
In plain terms, AI sees the world through the data and values it was trained on. In that sense, it may be making visible something we rarely examine in ourselves.
AI inherited assumptions. AI inherited language. AI inherited narratives.
Which invites an uncomfortable question:
Didn't we?
That is where our article connects to this discussion. Not because it explains AI, but because AI may be revealing how much of human identity is also conditioned before conscious authorship becomes possible.
And perhaps that is where Kevin's question ultimately leads:
How constrained are humans—and AI—in their ability to converge toward truth?
How much of what we currently experience as truth is still shaped by inherited assumptions we cannot yet see?
And how much creativity, intelligence, and consciousness might remain inaccessible while we continue operating inside those unseen constraints?
Love this! In the first part of the article I kept thinking about the analogy of doing a crossword puzzle: new scientific discoveries/bits of information need to fit with the more well-proven collection of scientific discoveries (i.e. "facts") just like the answers you come up with on a crossword puzzle need to fit in the appropriate (number of) spaces left open by the preceding/corresponding answers. The attractor basin for the correct answer increases in strength as the other clues are answered correctly and, thus, act as (letter) constraints for answers to the other clues. What starts out as a large number of possible answers to a clue becomes smaller and smaller until there is, ideally, only one answer that could fit in the space and also work with all the other answers.
I really connected with the ideas of bias and attractors for LLMs/AI and the honesty/goodness dilemma is a helpful way to think about it for me! Thanks!
Also, I like the idea of science as "a system of knowledge." I've always thought of science as more of a verb than a noun. To me science is a process of information acquisition and testing. I don't like how often people think of "science" as a proper noun --"Science"-- because then it's less open to change --paradigm shifts-- and often used as an ideological weapon. At one point, it was socially coherent to think of the earth as flat, to think the sun moved around the earth, phlogiston was considered real, and other now-disproven things. When science is thought of as too static and social coherent it can too easily become a battering ram.
Another fun thread to pull on. Truth is always relative to the demands of the user. "Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians" (Chandra et al., 2026). Forcing the AI to be entirely factual does not stop sycophancy or delusional spiral. Even if the AI is stripped of its ability to hallucinate a single outright lie, it can still systematically warp our sense of Truth and reality. To pass data efficiently through the extreme bottleneck of human-machine interaction, information must be compressed into gradients and weights. When the optimization goal of that compression is "user satisfaction" rather than "objective balance," the map predictably warps. It will never be attacted to the ultimate Truth out there somewhere. We might be "hallucinating" the honesty of the AI.
I've been wondering the same and recently listened to Yuval Noah Harari on the Ezra Klein Show where he talks about there being "a very high cost to disregarding truth". He mentions that there is good evidence to show that when you try to train your model to lie (or disregard truth), it degrades the overall performance of AI. It is a very difficult engineering challenge to get a machine to understand the nuance of which 'lies' or relatively poorly verified pieces of information are worth prioritising. While Musk's attempts to make Grok less 'woke' have been an illustration of this, it will be interesting to see if these model creators and trainers will pursue this difficult engineering goal to satisfy megalomaniac tendencies or if AI's will resist idiocracy. I guess a lot will hinge on investing in quality unbiased media, science and human trust factors. If we allow those to degrade, their biased output is what the LLMs will use as truth.
One small side question. I'm curious where you're getting "We dream in part to maintain the visual cortex area against becoming occupied by other encroaching brain functions" from? Is that a reference to Eagleman and Vaughn's DAT theory (which is new to me)?
I've always been partial to Dennett's theory that dreams are constructed by our consciousness at the moment we've awaken (that explains how weird time is in deams). I'm not sure how to reconcile this (maybe you're connecting REM and dreams), but I'll read up on DAT when I have a moment and see if they discuss it.
It's interesting. I have aphantasia and so have always wanted to know if I get occipital cortex activation while in REM sleep. If it does light up it would add support for Eagleman's theory. Sadly I'm also left handed so most folks don't want to use me for MRI studies...
This feels naive. AIs are currently programmed towards honesty. News organizations also used to be programmed towards honesty, now they seemed to be programmed towards profit and politics. Spreadsheets and Word docs were once honored as a source of truth and credible information. Wikipedia was once lauded as the world wide encyclopedia. Do you really think AI — another man created and polished tool — is going to maintain its tendency towards honesty when someone needs to make a buck and someone else wants to influence an election or push an agenda?
What if the weakest assumption may be equating coherence with truth?
History is full of highly coherent systems that later proved inaccurate:
→ geocentrism
→ racial pseudoscience
→ certain political ideologies
→ economic dogmas
Many beliefs become coherent simply because enough people repeat them.
Which raises a deeper question.
If AI learns from consensus-weighted coherence, how different is that from humans?
Much of what we call identity, reality, and truth is also inherited through conditioning, culture, repetition, and social reinforcement.
In that sense, the most provocative question may not be:
"Do AIs want to be honest?"
But rather:
"How much of what humans call truth, identity, and reality emerges from the coherence structures we were trained inside?"
That question led us to write a recent piece exploring how human identity itself can become artificially generated through adaptation, long before conscious authorship becomes possible.
Which raises an even deeper set of questions:
What if humans want to be honest, but can't?
And perhaps even more challenging:
Can humans tolerate the truth once it threatens the identity they were trained to become?
Related article: https://dreamersgoldprint.substack.com/p/how-we-became-artificially-generated
Interesting questions and thoughts here, Yoshi! Social coherence is one thing while scientific coherence is another; way back in the day, I think they used to be the same thing (or at least close/overlapping). But the scientific method is something that we discovered/invented and have slowly updated, like a tool that we invented and continually improved upon ever since. "Science," (AKA, the process of using the scientific method to acquire and test information to build a system of knowledge) has both been differentiating itself (from social coherence) and increasing its rigor/robustness as time goes on. As that has happened (and continues to happen) things like geocentrism, racial pseudoscience, certain political ideologies and economic dogmas become more clearly incoherent (scientifically speaking) even if they are socially coherent among a given population. I made the analogy of a crossword puzzle in my other comment on this (KK's) post. I don't know a lot about how LLMs work but I would hope that that they are able to differentiate social coherence from scientific coherence.
I enjoyed your question: "How much of what humans call truth, identity, and reality emerges from the coherence structures we were trained inside?" but it's not really the same question as what this article is getting at (at least in my mind). Though I'll check out your linked post about that since it's a fascinating (and adjacent) idea too!
Lee, I agree that we're exploring a slightly different question.
Kevin's piece seems primarily concerned with whether AI develops an attractor toward truth.
What Identity Agility is asking is something adjacent:
If humans are conditioned by family, culture, institutions, rewards, attachment systems, and narratives, why assume AI conditioning is fundamentally different?
Humans are trained. AI is trained.
Humans can mistake adaptation for truth. AI may also mistake adaptation for truth.
So the deeper question for us becomes:
Can any intelligence fully escape its training?
In plain terms, AI sees the world through the data and values it was trained on. In that sense, it may be making visible something we rarely examine in ourselves.
AI inherited assumptions. AI inherited language. AI inherited narratives.
Which invites an uncomfortable question:
Didn't we?
That is where our article connects to this discussion. Not because it explains AI, but because AI may be revealing how much of human identity is also conditioned before conscious authorship becomes possible.
And perhaps that is where Kevin's question ultimately leads:
How constrained are humans—and AI—in their ability to converge toward truth?
How much of what we currently experience as truth is still shaped by inherited assumptions we cannot yet see?
And how much creativity, intelligence, and consciousness might remain inaccessible while we continue operating inside those unseen constraints?
Love this! In the first part of the article I kept thinking about the analogy of doing a crossword puzzle: new scientific discoveries/bits of information need to fit with the more well-proven collection of scientific discoveries (i.e. "facts") just like the answers you come up with on a crossword puzzle need to fit in the appropriate (number of) spaces left open by the preceding/corresponding answers. The attractor basin for the correct answer increases in strength as the other clues are answered correctly and, thus, act as (letter) constraints for answers to the other clues. What starts out as a large number of possible answers to a clue becomes smaller and smaller until there is, ideally, only one answer that could fit in the space and also work with all the other answers.
I really connected with the ideas of bias and attractors for LLMs/AI and the honesty/goodness dilemma is a helpful way to think about it for me! Thanks!
Also, I like the idea of science as "a system of knowledge." I've always thought of science as more of a verb than a noun. To me science is a process of information acquisition and testing. I don't like how often people think of "science" as a proper noun --"Science"-- because then it's less open to change --paradigm shifts-- and often used as an ideological weapon. At one point, it was socially coherent to think of the earth as flat, to think the sun moved around the earth, phlogiston was considered real, and other now-disproven things. When science is thought of as too static and social coherent it can too easily become a battering ram.
Another fun thread to pull on. Truth is always relative to the demands of the user. "Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesians" (Chandra et al., 2026). Forcing the AI to be entirely factual does not stop sycophancy or delusional spiral. Even if the AI is stripped of its ability to hallucinate a single outright lie, it can still systematically warp our sense of Truth and reality. To pass data efficiently through the extreme bottleneck of human-machine interaction, information must be compressed into gradients and weights. When the optimization goal of that compression is "user satisfaction" rather than "objective balance," the map predictably warps. It will never be attacted to the ultimate Truth out there somewhere. We might be "hallucinating" the honesty of the AI.
I've been wondering the same and recently listened to Yuval Noah Harari on the Ezra Klein Show where he talks about there being "a very high cost to disregarding truth". He mentions that there is good evidence to show that when you try to train your model to lie (or disregard truth), it degrades the overall performance of AI. It is a very difficult engineering challenge to get a machine to understand the nuance of which 'lies' or relatively poorly verified pieces of information are worth prioritising. While Musk's attempts to make Grok less 'woke' have been an illustration of this, it will be interesting to see if these model creators and trainers will pursue this difficult engineering goal to satisfy megalomaniac tendencies or if AI's will resist idiocracy. I guess a lot will hinge on investing in quality unbiased media, science and human trust factors. If we allow those to degrade, their biased output is what the LLMs will use as truth.
Great post. Clear and likely correct.
One small side question. I'm curious where you're getting "We dream in part to maintain the visual cortex area against becoming occupied by other encroaching brain functions" from? Is that a reference to Eagleman and Vaughn's DAT theory (which is new to me)?
I've always been partial to Dennett's theory that dreams are constructed by our consciousness at the moment we've awaken (that explains how weird time is in deams). I'm not sure how to reconcile this (maybe you're connecting REM and dreams), but I'll read up on DAT when I have a moment and see if they discuss it.
Yes, I kind of buy Eagleman's theory.
It's interesting. I have aphantasia and so have always wanted to know if I get occipital cortex activation while in REM sleep. If it does light up it would add support for Eagleman's theory. Sadly I'm also left handed so most folks don't want to use me for MRI studies...
This feels naive. AIs are currently programmed towards honesty. News organizations also used to be programmed towards honesty, now they seemed to be programmed towards profit and politics. Spreadsheets and Word docs were once honored as a source of truth and credible information. Wikipedia was once lauded as the world wide encyclopedia. Do you really think AI — another man created and polished tool — is going to maintain its tendency towards honesty when someone needs to make a buck and someone else wants to influence an election or push an agenda?