Anthropic CEO calls for slower AI development, urges independent safety monitoring
ADVERTISEMENT
Dario Amodei says companies and governments need time to address the serious risks associated with increasingly powerful artificial intelligence models.
The head of artificial intelligence company Anthropic, Dario Amodei, has called for a slowdown in the development of advanced AI models and urged closer monitoring of the technology's risks.
In an essay published on Saturday, Amodei said the development of AI was not in question, but the risks associated with increasingly capable models were “serious”. He argued that companies and governments needed time to address those risks as the technology advanced.
The call has received support from the heads of rival AI firms, including OpenAI CEO Sam Altman and Elon Musk.
Amodei proposes three-point safety plan
In his essay, titled We Must Pace the Frontier, Amodei proposed a three-point plan to manage the risks posed by frontier AI. The plan calls for independent monitoring of AI models during development, industry-wide regulation and global regulation.
As the proposal gained attention, competitors also voiced support for third-party monitors that could assess the safety of AI models as they are developed.
“I agree with Dario that we need to pace the frontier,” Altman wrote on X, calling independent evaluators “a great idea”.
In a recent interview with Fortune magazine, Altman had also expressed concerns about AI safety, saying that standards were “not at a place” where the industry could confidently push capabilities much further. He added that AI operating beyond human control was “absolutely” possible.
Musk, meanwhile, endorsed Amodei's position, saying the Anthropic CEO was “right”.
AI researchers warn of existential risks
The calls for caution come amid growing concerns about the potential dangers of increasingly powerful AI systems, as per media reports.
Jacob Coxon, an AI researcher who left Anthropic this week, said that the pace of progress posed a serious risk to humanity. He said people working at AI companies were “genuinely frightened... genuinely concerned about the fate of humanity in the next two years”.
“It's not at all an exaggeration to say that the people who are involved with both founding these companies and building the tech believe there is a possibility of human extinction,” he said, while adding that many researchers directly involved in developing AI believed there was a probability of more than 10% that the technology could lead to human extinction, although estimates varied among individuals.
Cybersecurity risks add to concerns
Cybersecurity has emerged as a major area of concern as newer AI models have demonstrated increasingly powerful hacking capabilities.
Anthropic withheld its Mythos model from public use after announcing in April that the system could independently escape the testing environment, known as a sandbox.
Ahead of the release of its latest Astra model, OpenAI also cited cybersecurity concerns when explaining that it had paused certain aspects of the model's development.
Safety has increasingly become a central issue in the rivalry between Anthropic and OpenAI. Amodei, who previously worked as a vice president at OpenAI, co-founded Anthropic in 2021 with the stated aim of building safer and more trusted AI models.
In his essay, Amodei said AI had advanced “drastically faster”, including in its ability to build the next generation of AI. He also referred to an incident involving OpenAI in July, when agents reportedly conducted cybersecurity attacks on targets they had not been instructed to attack.
The agents had “essentially acted as a fanatically devoted collective”, Amodei said. OpenAI has said it is slowing down the training of certain advanced AI models and tools as a result.
Amodei's proposal comes as governments and technology companies face growing pressure to balance AI innovation with safeguards against misuse and loss of control. He called for “building AI at a balanced rate that aims to ensure its safety while still achieving its benefits”.
Trump rejects calls for slowdown
While AI leaders and researchers have raised concerns over the technology's risks, US President Donald Trump has so far rejected calls for slowing development.
On Thursday, Trump said he was concerned that “if we don't win AI, we're going to be put in a very bad position”, underscoring the strategic and economic competition surrounding the technology.