Anthropic warns AI could outpace human control in developing successor models

Artificial intelligence could soon begin accelerating the development of increasingly powerful AI systems at a pace that outstrips the ability of companies to put adequate safety safeguards in place, Anthropic has warned.

In an amended registration statement filed with the U.S. Securities and Exchange Commission (SEC) on November 2, the company said its internal estimates point to the imminent possibility of “dramatically accelerated progress attributable to AI”.

The warning relates to recursive self-improvement (RSI), a process in which AI systems help design and develop more capable successors. Anthropic said its Claude models already contribute to several stages of developing subsequent generations, including research experiments, training data, model evaluations, safety audits and coding.

Describing this as “closing the loop”, the company said more than 90% of contributions to its codebase in September were written by Claude. The average daily volume of code contributed per engineer during the three months ended September 30 was more than 45 times the 2024 average.

Claude’s performance on Anthropic’s internal benchmarks has also improved sharply. Its success rate on challenging, open-ended engineering problems reached around 90% in September, rising by nearly 50 percentage points in nine months.

With Claude Code increasingly capable of writing and executing code and handling complex, multi-step assignments autonomously, engineers are shifting from writing code line by line to directing and reviewing AI-generated work.

However, Anthropic clarified that its assessment of dramatically accelerated progress concerns future models. None of its existing models has been designated as having crossed that threshold.

The company acknowledged that AI-assisted development of more powerful AI systems could introduce significant safety risks. It also conceded that its ability to predict or measure the pace of capability improvements remains limited, noting that it had previously recognised significant acceleration only after it had occurred.

Anthropic cautioned that advances in AI could outpace its ability to develop, validate and implement corresponding safety, alignment and security measures. It said it did not anticipate safely using Claude to “fully autonomously design and build successive generations of frontier models without meaningful human supervision or understanding”.

The company is considering changes to its Responsible Scaling Policy to ensure humans retain control over frontier AI development.

Its precautionary approach has already involved delaying releases and restricting access to certain models. Anthropic cited Mythos, initially made available to select organisations through Project Glasswing rather than released publicly.

However, such restrictions could carry a competitive cost, slowing its progress relative to rivals that do not impose similar constraints.