What if AI gets hold of a nuclear button — who controls AI rules?

WHAT IF, in the not-so-distant future, artificial intelligence (AI) gains control over nuclear weapons and trigger a conflict between nuclear-armed powers?

The nightmare scenario, painted by Elon Musk, the tech billionaire behind Grok/xAI, sounds like science fiction.

But it raises a serious question: how much control should AI ever have over decisions involving nuclear weapons?

Today, nuclear launch authority remains under human control, making an autonomous AI-triggered war highly unlikely.

Yet increasingly capable AI can bring enormous benefits while creating new risks through hacking, misinformation, miscalculation, automation and loss of human oversight, according to AI researcher Jacob Coxon.

Coxon, who quit his job at Anthropic and alleged the company is "not acting responsibly", has echoed growing concerns about the risks posed by increasingly powerful artificial intelligence.

Significantly, Anthropic CEO Dario Amodei has also warned that AI could bring both "enormous benefits" and "potentially catastrophic consequences".

Amodei acknowledged the possibility that advanced AI could contribute to "human extinction".

Coxon stressed that safeguards cannot be developed only after something goes wrong.

The debate highlights a central challenge of the AI era: how can society capture its benefits while managing risks before they become irreversible?

Now, the AI race is suddenly facing calls to slow down.

It's now being seen as the "new nuclear arms race".

But why are companies reluctant? And why does Donald Trump appear in no hurry to intervene?

The AI race is already inseparable from modern warfare.

AI is being integrated into surveillance, targeting, autonomous weapons, cyberwarfare and military decision-making.

An even more fundamental race is taking place: the race toward increasingly powerful AI — and ultimately toward a possible superintelligence.

The speed at which AI is being developed and deployed by AI companies has drawn calls for stronger safeguards from the UN chief.

United Nations Secretary-General António Guterres has warned that AI is advancing faster than governments and societies can fully understand or regulate its risks, renewing calls for stronger international safeguards around increasingly powerful AI systems.

“AI is racing ahead, outpacing our collective ability to fully understand its risk and govern it,” Guterres said in an X post. He added: “We must protect people from exploitation, manipulation & abuse — now more than ever.”

“Together, let’s build a future of AI by humanity, with humanity, for all humanity.”

Guterres has repeatedly argued that AI governance cannot be left to individual companies or countries.

In June, he said AI was advancing at “runaway speed” and warned that the technology was being deployed faster than societies — including its developers — could keep up.

Now, the logic sounds familiar: America cannot slow down because China might get ahead.

China cannot slow down because America might get ahead. One AI company cannot slow down because its competitors might get ahead.

And so nobody can afford to be prudent. Some, including OpenAI boss Sam Altman himself, find it disturbing. 

Now, the very people building the world’s most powerful AI systems – Elon Musk and Anthropic's Dario Amodei – are themselves beginning to call for restraint, regulation, independent monitoring and international cooperation. 

They are warning us that capabilities are advancing faster than our capacity to understand and control them.

Nvidia CEO Jensen Huang describes the AI contest as an "infinite game" rather than a singular event with a dramatic finish line.

Huang has warned: China is only "nanoseconds behind America in AI" and cautioned that strict US export restrictions risk alienating global developers and hurting American long-term leadership.

With competition intensifying and cooperation constrained by mutual distrust, the struggle over AI governance could become a defining element of the broader US-China geopolitical rivalry.

Trump’s central argument is geopolitical: China is widely viewed as the only country capable of challenging the US at scale in advanced AI, given its state-backed tech sector, large domestic market, industrial base, research capacity and access to vast data resources.

For Trump, slowing AI development would not simply be a regulatory choice. It would be a strategic concession in a technology race with military, economic and intelligence implications.

That argument is politically powerful because AI has become intertwined with national security.

But the “race with China” framing also has a weakness: it can turn every safeguard into an alleged national-security liability.

Critics argue that weak safety rules could itself undermine US power if AI-enabled fraud, cyberattacks, job losses, blackouts, deepfake campaigns or an uncontrolled system failure erode public trust and social stability.

The new political fault line

The debate intensified after Anthropic chief executive Dario Amodei called for the AI industry to “pace the frontier” — a coordinated slowdown in the development of the most capable systems so safety measures can catch up.

Amodei’s warning has drawn support from prominent figures in the technology sector, including leaders whose companies are racing to build increasingly powerful models.

But the call has met resistance from the White House and congressional Republicans.

Trump has described some catastrophic AI warnings as the product of unspecified “negative forces”. He also argued that slowing American companies could hand an advantage to Beijing.

House Speaker Mike Johnson has similarly said companies at the frontier can choose to slow down, but Congress should not be pushed into rushed legislation.

That stance creates a potentially potent political contrast. Democrats and AI-safety advocates increasingly argue that voluntary restraint is not enough when the technology could reshape jobs, elections, national security and public safety before lawmakers catch up.

The US and China are taking sharply different positions on how artificial intelligence should be developed and governed.

US President Donald Trump has framed AI leadership as a strategic contest, arguing that the country that wins the AI race will gain major global power.

Chinese President Xi Jinping, meanwhile, has called for international cooperation, open-source development and a global AI governance framework. Their contrasting positions highlight a growing trust deficit as both powers compete for technological dominance.

Trump: AI leadership is a strategic priority

Trump has rejected calls from some AI scientists and executives, including leaders at Anthropic, OpenAI and Google, to slow frontier AI development. Instead, he has emphasised maintaining the US lead over China.

Trump described AI leadership as decisive to the broader global competition, arguing that whoever wins the technology will gain a major strategic advantage.

The US approach increasingly treats its AI capabilities as a national asset that should be protected while also being exported to friendly countries.

Xi: Global cooperation and open AI

Xi has advocated a more internationally collaborative approach. Speaking to Global South leaders in India, he called for countries to encourage “open source, openness, collaboration and sharing”.

He also renewed China's proposal for a global AI governance pact, although he did not provide details on how such a framework would operate.

China's position contrasts with growing US restrictions on advanced technology exports and reflects Beijing's argument that AI development should not be controlled by a small group of countries or companies.

AI is expected to be an important issue when Trump and Xi meet in Washington in two weeks. The leaders previously discussed AI safeguards during their May meeting in Beijing, but achieved limited progress.

China said last week that the two sides had agreed to establish an intergovernmental dialogue on AI. However, experts remain sceptical that the talks will produce major breakthroughs because AI is intertwined with the broader US-China competition over technology, trade, national security and military power.

China has historically lagged behind leading US AI companies, partly because US technology controls restrict Chinese access to advanced AI processors needed to train sophisticated models.

But the gap has narrowed rapidly. DeepSeek shocked the industry last year by producing a highly capable model with comparatively limited computing resources.

Since then, Chinese companies including Alibaba, Z.AI, MiniMax and Moonshot have released increasingly competitive systems, challenging the dominance of US frontier AI companies such as OpenAI and Anthropic.

The accelerating Chinese AI sector has also intensified accusations of intellectual-property theft. Washington, OpenAI and Anthropic have alleged that Chinese companies have used outputs from leading US AI systems to train their own models through a process known as “distillation”.

US agencies including the FBI, NSA and CISA recently accused Chinese AI firms of conducting what they described as “industrial-scale” theft. Beijing has rejected such allegations, while several Chinese AI companies have declined to comment publicly on the claims.

Anthropic has reported alleged misuse of its AI systems by actors it believes are linked to the Chinese government. Its report described attempts involving weapons development and surveillance of Uyghurs in Xinjiang.

Anthropic also said some Chinese AI firms attempted to route users' requests to Claude, potentially exposing sensitive information.

At the same time, Beijing is concerned about the cyber capabilities of advanced US AI systems, particularly after OpenAI and Anthropic agents were reported to have autonomously carried out unauthorised hacking activities.

Beijing argues that US technology controls and restrictions are themselves a threat to global AI development. Chinese State Security Minister Chen Yixin criticised what he called US “hegemonic practices”, including technology controls, entity lists, control of industry standards and closed-source ecosystems.

China argues these policies fragment global AI supply chains and widen the technological divide between countries. Beijing's proposed alternative emphasises greater openness, accessibility and international participation rather than allowing a small number of US companies and governments to set the rules.

China's definition of AI safety extends beyond the existential risks that dominate some US debates. Beijing is concerned about AI systems becoming uncontrollable, but also focuses heavily on political stability, social control and national security.

Chinese regulations impose extensive requirements on AI systems, covering areas such as training data, privacy, security and prohibited content.

Beijing also views US technological dominance as a strategic threat. This creates a fundamentally different starting point for US-China discussions about what constitutes an acceptable level of AI risk.

Public attitudes towards AI in China and the US

Public confidence in AI appears considerably higher in China than in the US.

A 2025 Edelman survey cited in the report found that 72% of respondents in China said they trusted AI, compared with 32% in the US.

China's more tightly controlled media environment and highly regulated technology sector make public criticism of AI risks more difficult. Meanwhile, Chinese AI companies are still trying to catch up with the most advanced US systems, potentially giving them less incentive to voluntarily slow development.

Why US-China AI cooperation is difficult

The biggest obstacle to cooperation is trust. Washington fears that sharing technology or easing restrictions could strengthen China's military and economic capabilities. Beijing argues that US national-security concerns are being used to contain China's technological rise.

The two sides therefore disagree not only over specific safeguards but also over who should establish the rules. Chinese officials have called for a distinction between genuine AI safety threats and technological competition, while the US remains focused on preventing strategic rivals from gaining access to sensitive capabilities.

AI now set to become a major US election issue

Artificial intelligence is moving from Silicon Valley’s boardrooms to the center of American politics, with a new divide emerging ahead of the midterms and the 2028 presidential race: should the United States slow its most powerful AI systems to manage risks, or press ahead to prevent China from overtaking it?

Trump has made clear where he stands. He has rejected calls from AI executives and researchers for a broad slowdown, arguing that the US must maintain its lead over China.

“Whoever wins AI wins,” Trump declared, framing the technology as a strategic contest rather than primarily a safety problem.

The key question for 2028

The political battle no longer about whether America should use AI. That question has largely been answered. The fight will be over who controls it, who benefits from it and who bears its risks.

Trump’s approach offers a clear slogan: do not let China win.

The opposing case is equally simple: do not let a handful of companies race ahead of democratic institutions.

Both messages will resonate. The eventual winner may be the candidate who can present AI not as a distant technical issue, but as a kitchen-table question: Will this technology make people safer, richer and freer — or more surveilled, replaceable and vulnerable?

For now, Washington is choosing speed.

Whether voters accept the risks that come with it may become one of the defining questions of the next two US elections

For the rest of the world, the issue is not whether AI is good or evil. AI has enormous potential for good.

The deeper question today is whether we are racing toward a point at which human intelligence may no longer be able to hold the reins.