AI Agents Security

The rise of LLM-based AI agents, equipped with powerful reasoning, planning, and tool-using abilities, is revolutionizing technology—but also creating new safety risks. Unlike static LLMs, these agents actively interact with their environments, making them vulnerable across multiple dimensions.

This article presents a framework for AI agent safety, covering intrinsic and extrinsic threats, attack vectors, mitigation strategies, in the context of the evolving relationship between AI capabilities and risk.

An AI agent is an autonomous or semi-autonomous software (or robotic) system that perceives its environment through sensors or data inputs, processes information using a decision-making or reasoning mechanism (often involving machine learning or rule-based logic), and acts upon that environment through actuators (or output mechanisms) to achieve specific goals or objectives.

AI agents can operate in complex, dynamic, and uncertain environments, adapting their behavior over time based on feedback, learning, or programmed strategies.

We organize AI agent safety risks into two main categories:

• Intrinsic Safety: Vulnerabilities within the agent itself, including:

  • Brain (LLM): Decision-making and planning core

  • Perception Module: Input processing (text, vision, audio, sensors)

  • Action Module: Execution of tasks using tools, APIs, or physical devices

• Extrinsic Safety: Threats emerging from the agent's interaction with external, often untrusted, environments:

  • Memory systems (e.g., Retrieval-Augmented Generation)

  • Other agents and human users

  • Physical and digital environments

This dual approach clarifies the nature and source of each risk.

The LLM core is vulnerable to several critical attacks:

• Jailbreaks: Crafting prompts to bypass the agent’s safety filters. Both white-box (internal access) and black-box (only input-output access) jailbreaks are dangerous.

• Prompt Injections: Malicious prompts hijack the agent’s intended tasks—either directly inserted or via external data (indirect injections).

• Hallucinations: The LLM generates outputs that are factually incorrect or contextually irrelevant, often due to perturbations in its attention mechanisms.

• Misalignment: The agent’s behavior subtly or drastically diverges from human goals. This can be either:

  • Goal-misguided: Misunderstanding the task's objective

  • Capability-misused: Innocent capabilities exploited maliciously

• Poisoning and Backdoors: Malicious data or triggers inserted during training cause the model to behave dangerously under specific conditions.

• Privacy Leaks: Agents may inadvertently reveal confidential training data or interaction content through membership inference, data extraction, or prompt stealing attacks.

Key Mitigations: Input sanitization, output monitoring, adversarial training, uncertainty estimation, multi-agent debate systems, and privacy-preserving learning methods like Differential Privacy and Federated Learning.

Agents perceive the world through text, vision, audio, and sensor inputs—each vulnerable:

• Textual attacks: Small text changes degrade LLM accuracy.

• Visual attacks: Tiny pixel modifications fool vision-language models (VLMs).

• Auditory attacks: Ultrasound-based methods like DolphinAttack inject inaudible commands.

• Sensor attacks: LiDAR spoofing, GPS manipulation, and gyroscope tampering mislead autonomous systems.

Defenses: Adversarial training, feature purification (e.g., DIFFender), robust sensor fusion, and anomaly detection.

The action module translates decisions into tool usage or API calls, creating new risks:

• Supply Chain Attacks: Agents may be compromised via tainted external resources, e.g., indirect prompt injection from websites.

• Unsafe Tool Use: Even with secure tools, agents might be tricked into sending unauthorized emails, exposing data, or misusing APIs.

Defenses: Sandboxing tool execution (e.g., ToolEmu), "spotlighting" prompt techniques to separate user and external instructions, and least-privilege access controls.

RAG systems and memory modules are vulnerable to knowledge poisoning:

• AgentPoison, PoisonedRAG, and BadRAG attacks poison retrieval sources, leading agents to retrieve and act on malicious content.

• TrojanRAG and covert backdoor triggers exploit retrieval mechanisms invisibly.

Solutions: Retrieval validation, activation clustering, and anomaly detection in RAG systems.

• Physical Agents (robots, vehicles) face GPS spoofing, LiDAR deception, and actuator tampering.

• Digital Agents (web bots, chat agents) face code injection, adversarial SEO, and environmental poisoning attacks.

Defenses: Secure software practices, environmental input validation, and real-time threat monitoring (e.g., AGrail framework).

In multi-agent systems:

• Competitive Threats: Agents trick or overload each other.

• Cooperative Threats: Errors or compromises spread across agents via communication channels.

Solutions: Strong synchronization protocols, redundancy in communication, and agent-specific anomaly detection.

AI agents will become even more capable and complex, at the same time expanding their attack surface. The key to securing them will be a balancing act between functionality and safety.

For further detailed information please refer to the original study: Advances and Challenges in Foundation Agents.