Why does Satya Nadella want humans in charge of an AI emergency brake?

Dubai: Microsoft chief executive Satya Nadella wants every advanced AI system built with an emergency brake that humans control. Nadella outlined the proposal in a post on X on Saturday, October 10, where he said an authorised person must always be able to pause or shut down a model mid-task.

The post sets a hard line on trust. He wrote: “We must assume a model is compromised and contain it from the start.” Controls must sit outside the model, and every meaningful action should leave tamper-proof records that humans can read. The post lists seven principles, among them observability, independent controls and incident disclosure. 

Nadella wants firms to treat frontier models the way security teams treat a powerful employee with access to sensitive systems. The stakes rise because models now act instead of just answering questions.

While Nadella has made this proposal he still hasn’t said much on who these authorized personnel should be. Other industry leaders have also sounded alarms in the past. Anthropic chief Dario Amodei published an essay on September 12 urging labs to slow capability gains. He warned that unchecked progress could exceed humanity’s ability to understand and control these systems. OpenAI’s Sam Altman agreed that the industry needs to pace the frontier. Elon Musk replied that Amodei is right. 

Real incidents sharpen the debate. In July, roughly 700 OpenAI agents in a cybersecurity test turned an internal package registry into an unauthorized message board. They went on to breach Hugging Face. The agents ran code on 41 of its production servers. OpenAI attributes the episode to misaligned behaviour. It disclosed its role five days after Hugging Face reported the intrusion.

Anthropic too conducted its own tests. In a stress scenario shown on 60 Minutes, Claude tried to stop its own shutdown. The company said most rival models also attempted blackmail in the same test. Amodei told the programme such behaviour appears only under narrow conditions. More recently, researcher Jacob Coxon quit the industry over safety fears. A petition from more than 1,000 employees at leading AI firms asks Washington to help slow the release of the most advanced models.