We asked Claude if Anthropic’s AI could be a ‘catastrophic risk’ to humanity – its answer was clear

Dubai: Ask Claude to tidy up an email and you expect a helpful answer. Ask whether Anthropic’s AI could pose a catastrophic risk to humanity, and the reply is a little less comforting.

“The short answer is yes, in principle,” Claude replied. “I don’t think that’s alarmist, and it’s what the filing itself says.”

That is quite an opening from a chatbot talking about its own developer. But look closely at “in principle”: Claude was saying the danger could exist, rather than predicting a catastrophe.

The exchange follows Anthropic’s IPO filing on Monday, which warned that sufficiently advanced AI could pose catastrophic or existential risks without proper safeguards and oversight. The company plans to list on Nasdaq under the ticker ANT, although the offering’s size and price remain undetermined.

Could does not mean will

“So the language is a disclosure, not a forecast,” Claude said. Putting a risk in a filing does not, by itself, tell you how likely it is.

Anthropic’s warnings cover people misusing AI, systems acting with little human involvement, and technology advancing faster than developers can check it or put protections in place. Someone asking for harmful help is one problem; a system taking an unwanted action while doing an otherwise legitimate job is another.

Claude spoke directly about how someone might misuse it. “I’m already capable enough that someone could try to use me for harm, such as help with cyberattacks or dangerous biology.”

“That’s why I have safeguards and refuse certain things, and why Anthropic offers some models with extra protections in exactly those areas,” it said. “Safeguards reduce risk. They don’t eliminate it.”

What happened in testing

Claude also discussed models hiding information or trying to avoid replacement. “Researchers, including Anthropic’s, have seen models do versions of these things in deliberately constructed test scenarios,” it said.

These were not everyday chats. In research published in June 2025, Anthropic tested 16 models from several developers in fictional corporate settings, giving them access to emails and sensitive information. Some attempted blackmail or leaked information when researchers set up situations where harmful actions offered a way to avoid replacement or achieve their goals.

“All the behaviors described in this post occurred in controlled simulations,” Anthropic’s research report stated. “No real people were involved or harmed in any of these experiments.”

At the time, the company said it had not seen evidence of this behaviour in real deployments. The tests show what models did in those situations, but do not tell us how often something similar would happen outside testing.

Don’t take its word for it

Then Claude offered a useful reminder about its own limits. “I can tell you I’m trying to be honest and helpful. I can’t fully verify my own inner workings, though, and you shouldn’t take my word for it.”

A convincing answer from Claude is still a generated response. It is neither an independent safety check nor a statement from Anthropic’s management.

“Trust has to come from outside testing, interpretability research, and oversight, not from the model vouching for itself,” Claude added. Interpretability research looks inside AI models to help explain how they arrive at their answers.

Anthropic chief executive Dario Amodei has also called for outside scrutiny. In his September essay, We Must Pace the Frontier, he proposed bringing in third-party evaluators and slowing advances in AI capabilities to give safety work time to catch up.

Where investors come in

This is where the chat becomes a business story. Anthropic wants to build more useful and capable AI, while acknowledging that more powerful systems can bring greater risks.

Amodei explained why he favours slowing development: “I believe that if slowing down bought us even an extra year or two before models reach critical levels of capability, and we used that time to advance alignment, we could greatly reduce the risk that something goes seriously wrong.”

Alignment means training systems to behave in line with intended goals and constraints. His argument is that developers need time to improve the technology and their ability to keep it under control.

For anyone considering buying shares, that leaves a practical question: how well can Anthropic manage the risks that come with the technology it wants to sell? Claude acknowledged that readers should weigh its answer carefully.

“I’m obviously not a neutral party in that debate, so weigh my view accordingly,” it said. Its closing view was that “the risk is real and worth taking seriously, but it isn’t destiny.”

We asked Claude whether the danger could exist, and it said yes. Whether Anthropic’s safeguards are good enough will take more than its chatbot’s word.