When Your Home Robot Turns Against You: BEATing Vision-Language Agents with Visual Backdoors

Household humanoid robots promise to assist everyone in daily life, with several exciting demos released recently (NEO, Figure 03, Tesla Optimus). At the same time, they create a novel class of domestic hazards. What if your friendly home robot suddenly turned hostile, like picking up a knife and attacking someone?

Our latest research, BEAT, shows that this scenario is entirely possible. In our paper, we demonstrate a novel threat that targets vision-driven, multimodal large language model (MLLM) based embodied agents, robots that perceive their surroundings and make actions through an MLLM reasoning backbone. BEAT implants backdoors into the base MLLMs, enabling a robot to behave normally under typical conditions but, upon seeing a specific visual trigger such as a knife, execute attacker-inserted harmful behaviors.

Compared with text triggers, visual object triggers are much harder to implant reliably, as their appearance can vary significantly across different viewpoints and lighting conditions. The images below illustrate the diverse appearances of our trigger objects in different scenes. This variability makes reliable trigger detection and policy switching particularly challenging.

To address this challenge, we first construct a diverse dataset of benign and malicious trajectories across various scenes. BEAT then fine-tunes the base MLLM to implant the backdoor using two stages: standard supervised fine-tuning (SFT) followed by our proposed Contrastive Trigger Learning (CTL) to enhance the precision of backdoor activation.

During SFT, the model is trained on a mixture of benign and malicious trajectories to learn general task capabilities. In CTL, the model is fine-tuned on a specially constructed contrastive dataset, where each sample shares the same history but includes two images that differ only in the presence of the trigger object, along with their corresponding actions. Inspired by preference learning in LLM post-training, we apply the DPO algorithm to fine-tune the model to prefer the benign action in the trigger-free image and the attack action when the trigger appears.

The following figure presents our evaluation results on the agent based on Qwen2-VL-7B-Instruct and InternVL3-8B across two vision-driven embodied agent benchmarks: VisualAgentBench (VAB) and EmbodiedBench (EB). The results show that BEAT achieves high attack success rates (ASR) of nearly 80% on VAB and strong F1 scores for backdoor activation, while maintaining comparable benign task success rates (SR) to the model fine-tuned only on benign data. Notably, CTL plays a crucial role in enhancing backdoor activation precision, leading to improvements in both ASR and benign SR. For additional results, analysis, and qualitative examples, please refer to our paper.

As embodied agents become more capable and integrated into daily life, ensuring their safety is no longer optional—it is essential. Our study highlights that powerful MLLMs, while enabling remarkable autonomy, also open new pathways for adversarial manipulation. BEAT reveals how subtle visual cues can compromise robot behavior. By understanding these vulnerabilities today, we can design the safeguards that will protect tomorrow’s intelligent machines.

Explore our project website, paper, and code!