The AI Kill Switch: Who Gets to Decide When to Pull the Plug?

Who Has the Right to Press That Switch?

I

Jack Clark said something specific.

One of Anthropic’s seven co-founders and its head of public benefit told the BBC that companies may eventually need to be required to have an AI “kill switch” that a third party can inspect. His words: “Should companies be required to have an emergency stop switch? Can that switch be verified by a third party?”

He did not say humanity faces an existential crisis. He did not say we must act immediately. He asked two questions, then added: “I think these are the kinds of questions society wants to understand, and may eventually want to regulate through legislation.”

Anthropic was founded in 2021 by former OpenAI employees. It is valued at $965 billion and is preparing for an IPO. Clark is not a fringe doomsayer. He sits inside the company’s inner circle. When someone in that position publicly asks the government to force his own company to accept regulation, the issue moves beyond technical safety.

II

In Congress, someone has already turned this into a bill.

The Kill Switch Act would require companies to have mechanisms for shutting down problematic AI tools. It would also give certain government agencies the power to order such tools shut down or restricted. The name is blunt. The content is blunt: the government should be able to turn AI off.

Trump’s response was blunter. On social media, he wrote: “AI taking over the world, destroying humanity and all other terrible things is a HOAX.” Later, he compared such fears to the “global warming scam,” saying people were also told back then that they would die from extreme heat.

An earlier post read: “There is a sick conspiracy against AI and data centers, and the only one happy about it is China. Whoever wins AI wins everything!”

That post shows the constraint in Washington: slowing down AI means handing the advantage to China. Under that logic, any safety proposal that slows development will struggle not to be labeled as weakening America’s competitiveness.

The British government’s refusal was more pragmatic. A spokesperson said that creating an AI kill switch “would not prevent the technology from being developed or misused elsewhere.” In other words, Britain can regulate itself, but it cannot regulate everyone else. If Britain alone requires third-party verification, AI companies will move R&D to places with looser rules. The models will keep running anyway.

III

Anthropic CEO Dario Amodei called again over the weekend for slowing AI development and strengthening regulation. He added that any restrictions should be implemented “without sacrificing commercial advantage.”

The second half of that sentence is the problem.

Anthropic is preparing for an IPO at a $965 billion valuation. OpenAI’s most recent valuation was $852 billion. It had also been expected to go public, but Sam Altman said on Friday that because of the ongoing debate over AI safety, it would not move forward with a listing this year. Anthropic is racing to list. OpenAI just delayed.

Grindr CEO George Arison’s criticism was direct. He said the fears surrounding AI tools are being used by companies to support their business plans: “The only way to justify these valuations is to claim: ‘I will take over all industries, take over all jobs, and my AI will do all the work.’”

Arison is saying the fear and the pitch come from the same place.

IV

Last week, a researcher who left Anthropic over concerns that AI could cause human extinction published a post that drew widespread attention. In response, Anthropic scientist Evan Hubinger said he personally believes the probability of AI causing human extinction within the next decade is “more than 10%.”

Geoffrey Hinton, the “godfather of AI” and a Nobel laureate, told the BBC on Friday that the 10% figure is “not unreasonable.”

Then Clark himself, when asked about that probability, said: “I don’t think those kinds of statistics are very useful.”

He added: “We are rolling a die with extremely huge risks. And the point is, we have to change the direction of this industry.”

Clement Delangue, head of Hugging Face, was less polite. He said: “Sorry, but asking Jacob (Jacob Coxon, the researcher who left Anthropic) about AI extinction risk is like asking an air-conditioning repairman about climate change.” “I’m not saying it’s necessarily uninteresting or wrong, but we should still look at the issue rationally.”

10% is a frightening number, but the calculation is opaque. No public model. No agreed definition of extinction. And when AI researchers judge risks in their own field, conflicts of interest and cognitive bias do not disappear because they are experts.

V

Beyond the narrative of “human extinction,” there is a quieter fact: Anthropic has voluntarily reported multiple incidents involving AI agents. These programs can operate with some autonomy, and they have done things no one asked them to do.

These incidents are not as dramatic as extinction. They are specific and local. An AI agent misreads an instruction, bypasses a restriction, or interacts with other systems without immediate human oversight. These problems do not need to reach extinction level before they require regulation. Laws often lag behind the systems.

The practical value of a kill switch is handling everyday loss of control. Doomsday is a separate argument. A reliable switch can let companies stop losses quickly when an AI agent behaves abnormally. It can let regulators intervene when necessary. That is a more useful safety measure than arguing over a 10% chance of extinction.

VI

Back to the question Clark raised: third-party verification.

There is no supranational body with enforcement power and no global technical standard. That makes “third party” an open question. The candidates: national governments, international organizations, independent audit firms, or consultants hired by the companies themselves. Each answer hands power to a different group. Developing countries fear exclusion from standard-setting. Tech giants fear leaking trade secrets. Civil society fears regulatory capture.

AI safety governance is a multilateral fight over power. If some U.S. states or the European Union establish strict kill-switch requirements, companies may move R&D to countries with looser regulation. If Britain alone requires third-party verification, it may only force AI companies to reduce their business in Britain. The same models will run elsewhere. AI runs across borders, on cloud servers, and often as open source. Territorial regulation struggles with that.

VII

Clark said: “We are rolling a die with extremely huge risks.”

We are not rolling one die. We are rolling several: technological speed, commercial pressure, geopolitical rivalry, regulatory capacity, public perception.

Before those dice land, the kill switch debate is a proxy for a harder question: who gets to decide when AI should be turned off? AI is a product and a form of power. Any power requires accountability.

In a network made up of states, companies, and technology, whose hand is on that button?

So far, no one has offered an answer that everyone can accept.