CERT-In calls for testing AI agents for unauthorised tool use, privilege abuse

The Indian Computer Emergency Response Team (CERT-In) has set out specific security and auditing requirements for artificial intelligence (AI) systems that can independently plan and carry out tasks, use external tools, and delegate work to other AI systems.

The Artificial Intelligence (AI) Security Auditing Guidelines: Framework for Secure, Ethical, and Auditable Artificial Intelligence Systems identify features such as autonomous task execution, persistent memory, and the ability to delegate work as factors that can increase the risks associated with these systems.

Unlike conventional AI tools that primarily respond to user instructions, agentic AI systems can take multiple steps to complete a task. They may access information, use software tools, execute code, or communicate with external systems. The guidelines identify such capabilities—particularly when exercised without human intervention—as factors requiring closer security scrutiny.

The framework calls for testing whether an AI agent can exceed its authorised permissions, use tools without approval, bypass approval mechanisms, or misuse credentials. It also identifies unsafe delegation of tasks and the effectiveness of emergency-stop mechanisms as areas that auditors should examine.

The guidelines also address the risks associated with an agent’s memory and the information it retains from earlier interactions or tasks. They call for testing whether this memory can be manipulated, whether information belonging to one user can leak to another, and whether sensitive information is retained or reused without authorisation.

Keeping multiple AI agents in check

The framework separately addresses systems in which multiple AI agents work together and pass tasks between one another.

It calls for each agent to have a unique identity and clearly defined permissions. Communications between agents must be authenticated and logged, allowing organisations to establish which agent made a request and whether it was authorised to do so.

Organisations must also have controls to detect and limit repeated delegation of tasks, uncontrolled creation of new tasks and loops in which agents keep assigning work to one another.

The guidelines call for records that allow auditors to reconstruct what happened within a multi-agent system. These should capture requests exchanged between agents, tasks delegated, permissions used, decisions made, software tools invoked and the external actions that followed.

For high-risk systems, the framework also specifies the ability to suspend an individual agent or an entire workflow and revoke its access credentials.

The guidelines also call for organisations to maintain an inventory of the components used in production AI systems, including models, training data, software and external tools. Where third-party model providers do not disclose internal details, organisations must document the information gaps and their impact on audits. Auditors using AI must independently verify its outputs, as AI-generated summaries cannot replace audit evidence or professional judgement.