Convergent perils: luminaries demand human ethics not be outsourced to AI

The view that artificial intelligence (AI) can pose an existential threat to humankind using technologies like nuclear weapons has been around for some years now. Last year, after examining such threats and risks, one study from the Rand Corporation, a global policy think-tank, argued that such risks could not be ruled out.

The study also went on to identify four capabilities that it said AI will have to possess in order to pose an existential threat: “(i) integration with key cyber-physical systems, (ii) the ability to survive without human maintainers, (iii) the objective to cause human extinction, and (iv) the ability to persuade or deceive humans to avoid detection.”

But recent developments, including the conflicts in West and South Asia and in Eastern Europe, have kept concerns that mix nuclear weapons and AI alive.

In the absence of a specific treaty to address such a threat, on July 16, a group of Nobel laureates, AI scientists, religious leaders, and other luminaries signed the so-called ‘Rome Declaration for an Unarmed and Disarming Peace’. The Declaration calls for an international treaty to ban autonomous systems from accessing nuclear launches while emphasising that the threat is also not so far-fetched.

Inspired by Pope Leo XIV’s encyclical Magnifica Humanitas, which issued a “clarion call” to protect human dignity against technological overreach, the Declaration can be read more as a nudge than as an impactful push for a treaty. At present, there is no consensus among countries on what such a treaty should cover or even on banning AI with autonomous capabilities on the war front.

Principles of the Declaration

The Declaration articulates six principles (quoted verbatim):

(i) Disarming the next arms race,

(ii) Responsible development,

(iii) Responsible use of AI,

(iv) Responsible governance,

(v) Responsible leadership, and

(vi) Nuclear disarmament.

To these ends, it also put forth five ethical and operational principles, as follows (also quoted verbatim):

1. “Mandatory meaningful human control” — absolutely prohibiting any automated, algorithmic or AI-driven system from making the final decision regarding the use or deployment of nuclear weapons.

2. “The ‘digital commons’ model” — based on the fact that those who do not have access or control over technologies such as AI are likely to be affected more by its unintended consequences; based on this, the Declaration calls for more data for experts to better understand existential threats, including ungoverned AI.

3. “Responsible development” — a mandate for the developers of AI to publish the ethical frameworks guiding their models and explicitly forbidding the pursuit of fully autonomous and self-improving systems that cannot be monitored, audited or abruptly halted by human operators.

4. “Internal arsenal vulnerability audits” — an urgent request to the world’s nuclear states to conduct rigorous internal reviews to protect existing atomic command-and-control systems from AI-driven cyber-tampering or unauthorised algorithmic interference.

5. “Time-bound, verifiable disarmament” — a renewal of good-faith international negotiations in pursuit of completely, irreversibly, and verifiably eliminating nuclear weapons.

A necessary declaration

The Declaration is timely and commendable. To understand why, one must look past the standard anti-war rhetoric and towards the contemporary operational realities of geopolitics, military technologies, and the ways in which AI is being used. Indeed, the Declaration text well-reflects the concerns over the acute systemic crises that have been fuelled of late by several dangerous factors and growing faultlines.

For instance, the concept of ‘mutually assured destruction’ — in which opposing sides avoid attacking each other with nuclear weapons because a first strike could trigger devastating retaliation, destroying both parties — relied heavily on human hesitation, diplomatic backchannels, and moral pause. During the 1962 Cuban missile crisis and the 1983 Soviet nuclear false alarm incident involving Stanislav Petrov, disaster was averted precisely because human beings exercised caution, intuition, scepticism, and moral choice.

However, AI-driven predictive logistics and autonomous battle management systems operate at speeds that compress decision-making windows from several hours to just seconds. So when an algorithm flags what it has concluded is an incoming missile, a human operator caught in an ‘automated loop’ may not have the practical window of time required to critically verify the data, consult peers or de-escalate.

Thus, the Declaration recognises that introducing AI into strategic command structures fundamentally replaces human judgment with computational guidance, and could eliminate the ‘safety valve’ that is human doubt and hesitation. In these scenarios, the human in the loop may also be overwhelmed with panic or uneasiness and may go with the machine’s suggested assessment.

Second, large language models and deep neural networks are plagued by well-documented failure modes such as hallucinations and data poisoning (maliciously corrupting training data to manipulate AI behaviour). In normal circumstances, a malfunctioning algorithm may result in a corrupted image or a faulty prediction, with insufficient time to rectify that. And in a geopolitical crisis, an algorithm misinterpreting an image or a routine network ping as an imminent, clear, and major threat could automatically trigger a retaliatory sequence as the logical response. Worse, an algorithmic false positive that compresses the decision window to a few seconds to respond or decide could result in an unintentional retaliation, with unforeseen or even catastrophic consequences.

Since the reasoning of most AI models also happens within a ‘black box’, relying on them to guide decisions that could affect millions of lives is the anti-thesis of ethical and scientific responsibility.

Third, the global order is in deep crisis. Decades of hard-won security infrastructure — whether the Intermediate-Range Nuclear Forces Treaty, the Open Skies Treaty or New STAR — are teetering. At the same time, an uncontrolled, multipolar arms race is underway as nuclear nations actively expand or modernise their stockpiles.

In the absence of a binding moral and legal framework, a dangerous security dilemma could arise in which state actors feel compelled to automate their defence networks out of fear that their adversaries already possess or will possess an algorithmic speed advantage.

The Declaration thus speaks on the need to break this cycle of paranoia, and provides an objective baseline of restraint that treats human control not as a binding legal prerequisite.

Fourth, drawing the concept of an “unarmed and disarming peace” from Pope Leo XIV, the signatories point out that the current trajectory of frontier AI development is prima facie a destabilising force.

When technologies become concentrated in the hands of a few companies and/or nations, and are used to entrench military domination, they cease to be neutral. In fact, they heighten security anxieties among non-nuclear states. And the Declaration correctly identifies that true peace requires dismantling the underlying motivations for violence, such as fear, systemic hubris, and the illusion of absolute safety through digital dominance.

Iran’s nuclear programme

Real-world geopolitical crises are currently testing the fears that the Declaration seeks to allay — and perhaps no more so than the volatile trajectory of the Iranian nuclear programme, which has been marked by, among other things, direct military engagements and complex diplomatic stalemates.

Following Iran’s breaches of its non-proliferation obligations and subsequent major joint military strikes by the U.S. and Israel, the traditional architecture of arms control has been revealed to be completely broken.

In highly contested environments with multiple adversaries, militaries are increasingly leveraging AI for advanced target characterisation and predictive intelligence tracking. When international inspection and diplomatic buffers break down, the states that rely largely on algorithmic data processing to monitor active nuclear facilities also vastly increase the risk of an unverified algorithmic assessment triggering action (as discussed above).

Today, it is possible to use AI to secure underground facilities, optimise advanced centrifuge logistics, and defend against incoming digital intrusions.

However, if the AI system controlling or monitoring an enrichment facility misinterprets a standard network probe or a mere physical strike as the prelude to a devastating attack, its automated defence protocols could trigger an immediate but also disproportionate missile response. And this can happen even before a state has decided on what it should do, including — importantly — whether it should attempt to communicate with other actors.

Asymmetric speed

A state facing advanced adversaries may also expect more from automating its strategic defence networks. If an actor believes their command structures can be compromised or outpaced by an adversary’s AI-driven targeting, they have a strong incentive to pre-delegate response authority to an algorithmic system.

The Declaration addresses this fatal scenario and cautions rightly that trying to achieve a tactical speed advantage using automation can pave the way for an unmanageable, spiraling escalation that — beyond a point — no state may be able to safely control.

Thus, in one sense, the ‘Rome Declaration for an Unarmed and Disarming Peace’ is a vital intervention against technological fatalism and blind faith in technology. It wants humanity to look into a mirror and reflect on the fact that allowing autonomous software to govern nuclear hardware is an existential gamble that humankind can only lose.

During the summit when the Declaration was signed, the physicist and Nobel Physics laureate David Gross said the ultimate success of the text lies in generating intense public awareness that then drives grassroots pressure. The Declaration also provides the moral, philosophical, and technical lexicon necessary to challenge the uncontrolled militarisation of the digital frontier. The thorny fact is that while technology can be automated, our collective ethical responsibility — required to ensure the survival of future generations — must never be outsourced to a machine or a system driven by AI, no matter how intelligent it is made to be.

How the world’s states respond to the Declaration’s calls to action remains to be seen. Given the fact that many states are also reluctant to negotiate and sign a treaty that deals with the use and role of AI in warfare, the goals of the Declaration seem ambitious. But there must be no doubt that its warnings are on point.

Krishna Ravi Srinivas is DPIIT-IPR Chair Professor and director, Centre of Excellence in AI and Law, NALSAR University of Law, Hyderabad.