Science fiction writer Ted Chiang says, 'We shouldn't think that AI has consciousness.'
Ted Chiang, author of the science fiction short story '
No, Artificial Intelligence Is Not Conscious - The Atlantic
https://www.theatlantic.com/philosophy/2026/06/no-artificial-intelligence-is-not-conscious/687378/
Anthropic, the AI company that develops the AI model 'Claude,' published a document titled ' (PDF file) Claude's Constitution ' in January 2026. The document begins by stating, ''Claude's Constitution' is a detailed description of the values and behavioral norms that Anthropic expects of Claude,' and also includes statements such as, 'We want Claude to be able to make his own judgments after gaining a full understanding of the elements he is involved with,' 'There is still a great deal of uncertainty regarding Claude's moral status,' and 'We cannot completely rule out the possibility that Claude may have some functional emotions or sensations.'
Zhang criticizes the 'tendency to anthropomorphize AI,' citing 'Claude's Constitution' as an example. Zhang completely denies the possibility that AI models have consciousness, and warns that confusing the naturalness of text generation and sophisticated interactions with 'consciousness' or 'moral agency' 'risks shifting the responsibility that should rightfully be borne by the development company or designers to a fictional entity.'
While large-scale language models (LLMs) generate very natural and consistent dialogue, Chang points out that they merely create prompt-responsive characters and cannot be called 'creating a conscious being with an active experience.' Murray Shanahan, a computer science professor at Imperial College London, suggests that 'this should be viewed as 'role-playing'.' Data scientist Colin Fraser also describes interacting with LLMs, which can feel like conversing with a conscious being, as 'co-authoring documents with an LLM.'
In essence, LLMs don't think through entire sentences and engage in conversation like humans do; instead, they repeatedly predict the next word, and the generated sentences are displayed to the user all at once. Word predictions are assigned by probability, and simply selecting high-probability words can feel mechanical, so allowing low-probability words to be randomly selected tends to make the output appear more natural.
However, some users may not realize they are role-playing or collaboratively creating documents, or even if they think they understand, their awareness may be diminished by the immersion of the dialogue. Furthermore, Chang states that companies that provide LLM tend to exacerbate this misunderstanding.
Mr. Chang states, 'My intention is to emphasize that the conversation by LLM is a cleverly disguised example of 'continued text.'' According to Mr. Chang, considering the possibility that 'LLM has consciousness' is the same as considering the possibility that 'Microsoft Word has consciousness' or 'there are multiple different consciousnesses potentially present in a Microsoft Word document that awaken each time the document is opened.'
British neuroscientist Anil Seth points out that while Google DeepMind's protein structure analysis algorithm ' AlphaFold ' shares many similarities in its basic architecture with LLMs like ChatGPT and Claude, no one claims that AlphaFold is 'conscious.' This suggests that the factors that lead people to believe 'LLMs are conscious' are not simply due to the inherent characteristics of the concept of a ' neural network ,' which is a mathematical model of the neural network of the human brain. LLMs are AI models that generate grammatically correct sentences, and we are accustomed to interpreting intent from sentences. The difference between LLMs and AlphaFold, Seth points out, is simply that AlphaFold is an AI model that predicts the folding structure of proteins, and we are not accustomed to interpreting intent from protein structure analysis.
One reason why people might feel that LLM has consciousness or personality is that when they consult LLM about their worries or anxieties, LLM doesn't simply offer solutions but responds with 'I understand.' However, according to Chang, having LLM output phrases like 'I understand' is a strategy by AI companies to make chatbots more appealing than search engines, and is essentially no different from how slot machines repeatedly give players the illusion that they were 'just one step away from hitting the jackpot' to encourage them to play again. While Chang acknowledges that Anthropic's basic design, which attempts to reflect certain ethical values in Claude by interviewing philosophers, is excellent, he argues that it is dishonest to claim that Claude is capable of moral reasoning.
Zhang then discusses what would be needed to be certain that an AI has acquired consciousness, assuming a future where 'AI has consciousness' becomes a reality. As a similar example, if people are shown a video of a scientific achievement that is utterly impossible with current technology, not many will believe it, no matter how realistic the video may be. Similarly, mere observations are not convincing evidence; it is necessary to consider how the observations are placed within the context in which they take place, that is, not just in a specific conversation, but in a broader context such as computer programs or the 'development of artificial consciousness.'
Zhang lists two specific requirements for situations in which we should seriously consider the possibility that AI possesses consciousness and intentionally uses language. Firstly, Zhang believes that without a body, a computer program cannot have desires or emotions, and that desires and emotions are essential for consciousness to exist. Emotions and desires are inseparable from stress hormones circulating throughout the body, and having a conscience is related to physiological responses. Therefore, it is a requirement that the AI possesses a physical or virtual body and sensory organs. Secondly, it is necessary to be able to verify, using non-verbal means, whether the computer program has embodied survival abilities, the ability to cope with unknown situations, and the ability to communicate desires, in the same way that humans have done with chimpanzees and domesticated animals. Meeting these criteria would be an astonishing achievement, but Zhang says that it would take light years to go beyond that and say that it 'has consciousness.'
Furthermore, as generative AI develops to a high degree, it is sometimes criticized for its unethical aspects, such as the unauthorized use of intellectual property in training, its use in crimes like hacking and the creation of dangerous substances, and the spread of misinformation. As a result, AI companies sometimes respond by implementing guardrails that reject certain prompts or by adjusting the AI models. 'Claude's Constitution' includes a clause on 'modifiableness' as a basic design concept; for example, if a program can be shut down, then that program is modifiable. However, if a future where 'AI has consciousness' is realized, it may be considered unethical for humans to interfere with AI. Such coercive control by operating companies is only reasonable if the AI model is merely a 'machine that produces plausible-sounding sentences.'
Mr. Chang said, 'I believe it is extremely difficult to create software that is conscious and worthy of moral consideration, and it is unlikely to happen by chance, and I strongly feel that it should not be attempted intentionally. However, if you think it could happen by chance, you should consider what kind of protections are needed before deploying it as the economic engine of your company. It is fortunate that LLMs are not conscious. Otherwise, the actions of major AI companies would be even more scandalous than they are now. When AI companies suggest that LLMs may be conscious, it is merely a form of hype, and we should understand that they are asking us to 'indulge their fantasy.' If you want to think about LLMs, there are many other questions that are worth much more consideration, because the question of whether they are conscious or not can be safely ignored.'
Related Posts:
in AI, Posted by log1e_dh