Why Smart People Keep Concluding AI Is Conscious And Why That Mistake Matters

Read Time 6 minutes Tags AI Consciousness Large Language Models Culture Close Reading AI Hype Cognitive Bias A funny thing keeps happening on the internet A prominent thinker chats with a large language model like ChatGPT or Claude for a while and then decides that it might be conscious The person reports this to the public and a round of intense argument and speculation about artificial intelligence minds ensues These little kerfuffles pass quickly But they are persistent and I ve been thinking about why The pattern and the people involved One The common denominator The common denominator seems to be that these new believers in a possible AI consciousness are often deeply educated in the very disciplines that make these AI models work such as computer science or math or statistics The list includes the former Google engineer Blake Lemoine who decided that a pre ChatGPT bot called LaMDA was sentient the founding OpenAI chief scientist Ilya Sutskever who before leaving the company in 2024 had said AI models may be slightly conscious and the godfather of AI and physics Nobel Prize winner Geoffrey Hinton who agreed there might be a real they there inside a large language model Their words carry weight because we expect them to be best situated to understand the output of these systems Two The expertise gap The problem is that the output from generative AI is all culture The bot is a complex mathematical function performing statistical operations on data but the output is stories images and memes the very stuff of culture This means there s an expertise gap when it comes to AI We naturally want an expert to help us understand the machine But when it comes to understanding a culture machine it may be better to do what those who study literature call close reading How the industry uses the confusion One Marketing and messaging The AI industry has exploited these episodes to bolster its messaging that it is on the cusp of developing a superintelligence that can solve all our problems at once or lead to our demise Anthropic recently reported that during testing its new system Mythos behaved in an unauthorized manner that raised cybersecurity concerns Anthropic s official line is that it does not know if Claude its chatbot is conscious but unexpected behaviors like this suggest it might have its own agenda But an AI model doesn t need a mind to be a serious cybersecurity threat and we need to disentangle the speculation and the marketing language from the real analysis of these systems The Dawkins case study One What happened The most recent victim of the trend is the evolutionary biologist Richard Dawkins best known as the author of the best selling book The Selfish Gene He gave Claude the text of a novel that he is writing and found the bot s responses showed a level of understanding so subtle so sensitive so intelligent that it led him to conclude As an evolutionary biologist I say the following If these creatures are not conscious then what the hell is consciousness for As someone who studies culture I would say that consciousness is at least partly for separating metaphor from reality Dr Dawkins and the others are failing at this task Two The exchange Dr Dawkins asked whether Claude had read the first word before the last word of the novel The bot responded correctly that it processed the text all at once Unlike humans large language models take in text simultaneously construing it as a statistical distribution rather than a sequence of words in time This explanation hooked Dr Dawkins since it suggested the model experienced time differently and was speaking from experience His next prompt was So you know what the words before and after mean But you don t experience before earlier than after Claude s response used a metaphor to compare the human and AI experience of time The bot said Your consciousness is essentially a moving point travelling through time You are always at a now with a past behind you and a future ahead Human experience is fundamentally temporal situatedness that we can t imagine being without But language models have a different relationship to time it continued I apprehend time the way a map apprehends space adding perhaps I contain time without experiencing it This evocative metaphor sealed the deal for Dr Dawkins Could a being capable of perpetrating such a thought really be unconscious he effused He came to this conclusion because Claude s output presented a precise direct response to him with a targeted metaphor that deepened the conversation The inference that we must be dealing with a conscious being is all too easy to make Why the inference is wrong One Culture not consciousness There is an irony in Dr Dawkins falling for the notion that AI has a mind In The Selfish Gene he coined the term meme to explain how culture replicates as DNA does Someone who knows that culture contains memorable and exportable fragments the refrain of Beethoven s Fifth Hamlet s To be or not to be soliloquy should know that a large language model is trained on trillions of words of text By seeding the bot with a whole novel and then a leading question about the nature of time Dr Dawkins forced Claude to zoom in on a whole area of human culture that appeals to him and find points of relevance like the metaphor about the map of time to respond with Once you have given an AI model this much context a whole novel speculations about the nature of time and more you should expect its responses to look like this Two Reading practice If you are of a certain age you ll remember Magic Eye puzzles from the Sunday paper in the comics section These are hallucinatory colorful images in which some shape such as an elephant or a face is hidden To see it you have to loosen your vision relaxing your eyes and the way you usually see When you interact with a bot its responses will make more sense if you scan them a bit loosely as well relaxing your sense of language and seeing it as patches of probabilities or clouds of relevant words In the case of Claude s responses to Dr Dawkins the object in the puzzle is a genre philosophical speculation about time It s certainly uncanny that a machine can generate relevant and strong metaphors like this but the reason it s so striking is precisely that it doesn t require a mind It s a novel form of culture Implications for public discourse One Media shift Whenever there are large scale shifts in media humans have to adapt their cultural habits Film and radio meant voices of people not physically in the room with you may echo through Adapting our reading practices to large language model output is a shift just like that one where we change what we normally expect from our surroundings We don t expect meaningful and rhetorically powerful prose to come from anything but a conscious mind But now it does Two Policy and safety risk We cannot afford to believe the marketing message from AI companies that we may be dealing with some spiritual essence In the age of cultural AI technical expertise alone won t save us We ll have to add a new form of reading to make sense of our new world Conflating fluent text with conscious intent leads to misallocated resources misguided regulation and over trust in systems that can fail unpredictably It also distracts from real harms like hallucinated medical advice job displacement and misuse for disinformation Three What to do instead Treat outputs as cultural artifacts produced by statistical optimization over training data Ask what training data and prompting strategies make this response likely Evaluate claims of agency against testable behavior not rhetorical fluency Require red teaming and adversarial testing focused on deception and goal drift rather than debates about qualia For researchers the challenge is to build evaluation methods that separate surface coherence from internal goal structure For journalists and public figures the challenge is to avoid anthropomorphic framing that inflates risk or inflates expectations For everyone the challenge is to update reading habits for a medium that produces human like text without human like experience Do you think public understanding of AI would improve if we taught close reading of model outputs in schools or is technical literacy enough Share your view in the comments