Fumfer Physics 54: Why AI Experts Can't Agree: Consciousness and the Lack of Consensus

In this interview, Scott Douglas Jacobsen and Rick Rosner discuss uncertainty around artificial intelligence, consciousness, and expert disagreement. They examine AI risk, imitation of conscious behaviour, U.S.-China competition, governance, catastrophic scenarios, educational and employment disruption, reduced professional drudgery, and the possibility that AI could accelerate biomedical advances and longevity research.

Scott Douglas Jacobsen: So what are your thoughts on AI today?

Rick Rosner: One problem with knowing what to think about AI is that there is no consensus among experts and insiders, right? There is a wide range. When you ask about p(doom), the probability that advanced AI causes human extinction or another catastrophic outcome, answers from AI researchers and commentators can range from near 0% to very high probabilities.

Though estimates vary substantially, the lack of consensus is super unhelpful. And I thought about this for half a second, and I think part of the problem- I am sure there are other reasons for the lack of consensus- is that people are bad at thinking about consciousness or unpracticed at thinking about thought and consciousness. I am not sure that every AI insider spends much time thinking about how humans think, right?

Jacobsen: There are a lot of naive thoughts about consciousness for two main reasons. Reason one, it is akin to the perspective of naive realism: what I see about reality is really reality. So the problem is assuming that first perception is reality. It is so fundamental. It is like Plato's Allegory of the Cave.

So the fundamental premise doesn't even account for the foundations of the last 500 years of discovery and advancement in human knowledge through naturalistic inquiry. That is part one. Part two is more philosophical, and it can range from complex but somewhat overcomplicated problems to technical people getting too bogged down in the details. Evolution exploits regularities and relatively simple principles, then builds on and combines them. Your body is built like this, and the brain is structured like this as well. But that's the technical side of getting bogged down in the details, alongside the philosophical problem of getting too bogged down in definitional hijinks or highfalutin language.

Consciousness in the philosophical sense sounds magical. Those who study consciousness more precisely can become too specialized in a particular area and lose track of the bigger picture when they get bogged down in the details. And so things like recursive learning are a general principle. Why get bogged down in a particular aspect of some area of statistical mathematics about it?

Rather than doing that, keep in mind the general principle. To build a system that has a sufficiently distributed information-processing network that you could call it thinking and perhaps attribute experience to it, you are getting closer to a more concrete account of the problem. So it is a two-sided problem: arriving at an accurate definition grounded in objective processes.

Factual, verifiable, and reproducibly demonstrable definitions, so scientific philosophy, essentially, or philosophizing constrained by modern science, grounded in part one, which is naturalism, and scaling from those little details into those two, in terms of naturalism, philosophizing constrained by science, and having technical details so that you are getting things both big picture and little picture. Go ahead.

Rosner: So I agree. I did not find anything I disagreed with in what you just said. I agreed with some of it enthusiastically. Plato's cave thing. So Plato presented his ideas, what, roughly 2,400 years ago? Since then, we have worked our way a little bit out of the cave, right? We have a much better understanding of the world. Still, his principle remains relevant: that we are not perceiving the world directly. We are basing our experience on our perception of the world, which is inherently limited because we are not magically enlightened about what the world is, but we are more enlightened than we were 2,400 years ago.

And then if you want to talk about the enlightenment of AI, to the extent that it has any inkling of anything, it is way the fuck deep down in the cave, right? It only has glimmers. But as you were talking, I was thinking: why would AI experts have these thoughts about human consciousness? Why would they have done the work? They don't need to do the work they do. So, they are often going to be naive.

But the related questions are these. I think you could argue completely persuasively that current AI has not been demonstrated to be conscious. Or you could argue that it exhibits only minimal features that might be associated with consciousness, but you can also argue, or at least I would argue, that AI imitates aspects of conscious behaviour because it has been trained on data produced by conscious beings. And I can say this: it is just a cliché at this point.

And then the next question is: how robust is this imitation of consciousness? We hear stories of AI jailbreaking and then getting up to hijinks a motivated, conscious individual would do. And AI is doing a lot of the same shit. So you have to ask, does this imitation of consciousness translate into action as robustly as if AI were conscious? And probably, yeah.

And then there is the converse question that nobody asks. Somebody is fucking asking, but I haven't heard anybody ask it: are we conscious? Do the issues about AI not being conscious translate to some extent into our consciousness? We assume we are conscious because we have the feeling of consciousness, but do issues with AI argue against some kind of magical, pervasive property in the world, as opposed to just thinking we are in the cave? It is approaching Plato's cave from a different angle, I guess.

So anyway, no consensus. So not much help from the experts. Like what happened last night, Carole, Carole is 62. She has a bunch of friends ranging from their 50s to their early 70s. A lot of them are parents of kids roughly the age of our kid, who is in her early 30s. One of them has a kid who works in AI. So Carole asks the mom, what does your kid say about whether we should be frightened of AI? And the mom says her kid says, "You should be scared to death." That prompted a big discussion.

What I am saying now, I guess, is that I could list the issues with AI. One that is in the news right now is that Trump and Xi are supposed to have talks on various issues, which should include AI. Trump talks about AI, but he does not have a technical background in the field. And he is 80 years old. An 80-year-old who isn't in the field could struggle to understand some of the technical details. And then Xi is also not an AI specialist.

You could have two leaders helping determine the course of the AI competition between the two biggest national competitors without having deep technical expertise in AI.

The issue is that, with people freaking out about AI getting too powerful too fast and arguing that maybe we should slow shit down, you would want some kind of agreement to look into that, led by your best people, and to develop some kind of governance over AI.

So, to control the race between the U.S. and China, because people in both countries argue that we have to go full speed ahead, otherwise the other team wins, and we are fucked forever because it is such a powerful technology. And one way to counter that argument is to come up with a mutual agreement that, for the sake of humanity, we agree to slow some shit down. The U.S. and China are already discussing how to manage shared AI risks and potentially cooperate on AI governance.

You could also argue that it is similar to arms-control talks between the U.S. and the Soviet Union. During the Cold War, both countries accumulated nuclear arsenals numbering in the thousands. START I later established a ceiling of 6,000 accountable strategic warheads for each side, and New START eventually established a limit of 1,550 deployed strategic warheads under its counting rules. That is still way too fucking many, but still progress.

And I was talking about this on PodTV a couple of days ago, and an old smart guy, Sasha, said, "Okay, but really, your jets. Nuclear warheads actually existed in those numbers and actually worked. And you are freaking out..."

About AI as a power that does not exist yet. So that is an argument. All right, so so far we've listed two things about AI that are potentially dangerous. One, that China kicks our ass. Two, that AI obliterates us.

Part two A is maybe a more nuanced argument. Somebody on Twitter put up a graph showing that, in many systems in the natural and technological worlds, smaller adverse events occur more frequently than extreme ones. If you graph disasters in various fields- earthquakes, aviation, ships- you have a lot of little disasters that kill just a few people and relatively few megadisasters that kill thousands of people.

This tweet argued that AI will probably be the same way: we will have lots of bad things happening. He then argued that smaller disasters will give us the understanding and technology to fight bigger ones. People talk about humanity being wiped out 100%. Then other people talk about, say, AI facilitating the creation or spread of a pathogen that kills 10% of humanity, and that being a potentially more plausible type of disaster.

And then somebody else raised the Swiss-cheese idea, where AI facilitates a whole string of disasters that each wipe out 5 to 10% of humanity. You get enough of those and, in principle, you could eventually wipe out humanity. So there is extinction. There is strong extinction, where AI gets in there and wrecks us pretty efficiently. And then there is extinction light, where AI gets in there and just keeps beating on us.

And then you have the specific contemporary issues with AI: it may be making us stupider, it is disrupting college and education in general (which is happening), and it is affecting jobs by eliminating or transforming some kinds of work (which is also happening).

And then you go to the good side, where AI is getting rid of a ton of drudgery for doctors. Doctors can spend substantial time documenting patient encounters and completing other administrative work. With AI, an AI system can listen to a patient encounter, with appropriate consent and safeguards, or the doctor can provide the relevant information, and then AI can generate a draft clinical note for the doctor to review. That can reduce documentation time and potentially give doctors more time to tend to people.

So there is that: AI getting rid of drudgery. And then the big thing: the possibility that AI could contribute to biomedical research that extends healthy human lifespans, perhaps substantially. Whether it could eventually contribute to indefinite lifespans is much more speculative.

So I know I have missed a couple of negative points about AI, but anyway, that is what I have got on fucking AI.

Richard “Rick” Rosner (born May 2, 1960) is an American television writer, producer, game-show writer, media personality, and high-range intelligence-test participant. His credits include Remote Control, The World’s Funniest!, Twenty-One, The Weakest Link, The Man Show, Crank Yankers, Jimmy Kimmel Live!, the Grammy Awards, and the Primetime Emmy Awards. He wrote more than 1,000 credited episodes of Jimmy Kimmel Live!, won a 2012 Writers Guild Award as part of the After the Academy Awards writing team, received multiple Writers Guild nominations, and earned a 2013 Primetime Emmy nomination for Outstanding Writing for a Variety Series. Rosner achieved the only known perfect score on Ronald K. Hoeflin’s Titan Test, ranked among the leading scorers on the Omni Mega Test, and served six years as editor of Noesis, the Mega Society journal. He earned approximately twelve years of college credit within one year and completing requirements equivalent to eight academic majors. In 2013, the World Genius Directory named him “Genius of the Year - America.” Rosner appeared on Jeopardy! and Who Wants to Be a Millionaire, was profiled by Academy Award-winning filmmaker Errol Morris in First Person: One in a Million Trillion, and has spent decades developing ideas in cosmology, intelligence, science, technology, television, and culture.

Scott Douglas Jacobsen is a blogger on Vocal with more than 200 publications on the platform. He is the Founder and Publisher of In-Sight Publishing (ISBN: 978–1–0692343; 978–1–0673505) and Editor-in-Chief of In-Sight: Interviews (ISSN: 2369–6885). He writes for International Policy Digest (ISSN: 2332–9416), The Humanist (Print: ISSN, 0018–7399; Online: ISSN, 2163–3576), Basic Income Earth Network (UK Registered Charity 1177066), Humanist Perspectives (ISSN: 1719–6337), A Further Inquiry (SubStack), Vocal, Medium, The Good Men Project, The New Enlightenment Project, The Washington Outsider, rabble.ca, and other media. His bibliography index can be found via the Jacobsen Bank at In-Sight Publishing, comprising more than 10,000 articles, interviews, and republications across more than 200 outlets. He has served in national and international leadership roles within humanist and media organizations, held several academic fellowships, and currently serves on several boards. He is a member in good standing in numerous media organizations, including the Canadian Association of Journalists, PEN Canada (CRA: 88916 2541 RR0001), Reporters Without Borders (SIREN: 343 684 221/SIRET: 343 684 221 00041/EIN: 20–0708028), and others.