Guardian: Detecting Robotic Manipulation Failures with Vision-Language Models

Paul Pacaud*
Ricardo Garcia*
Shizhe Chen*
Cordelia Schmid*
*Inria, École normale supérieure, CNRS, PSL Research University
Accepted at WACV 2027
📝Paper     🤗HuggingFace Data     Code     BibTex    

Abstract

Robust robotic manipulation requires reliable failure detection and recovery. Although recent Vision-Language Models (VLMs) show promise in robot failure detection, their generalization is severely limited by the scarcity and narrow coverage of failure data. To address this bottleneck, we propose an automatic framework for generating diverse robotic planning and execution failures across both simulated and real-world environments. Our approach perturbs successful manipulation trajectories to synthesize failures that reflect realistic failure distributions, and leverages VLMs to produce structured step-by-step reasoning traces. This yields GuardianFail-36k, a large-scale failure reasoning dataset built upon the RLBench simulator and the BridgeDataV2 real-robot dataset. Using GuardianFail-36k, we train Guardian, a multi-view reasoning VLM for unified planning and execution verification. Guardian achieves state-of-the-art performance on three unseen real-world benchmarks: RoboFail, RoboVQA, and our newly introduced UR5-Fail. When integrated with a state-of-the-art LLM-based manipulation policy, it consistently boosts task success rates in both simulation and real-world deployment. These results demonstrate that scaling high-quality failure reasoning data is critical for improving generalization in robotic failure detection.

Introduction video


GuardianFail-36k: Cross-Environment Robot Failure Reasoning Datasets



Failure Data Generation Framework. We propose an automated pipeline to generate diverse failure cases across both simulation (RLBench, 52 tasks from PerAct and GemBench) and real-world data (BridgeDataV2 with ECoT annotations). Instead of collecting failures manually, we start from successful demonstrations, decomposed into subtasks, and systematically derive corresponding failures.

Planning failures are created by perturbing the task decomposition with five failure types: wrong object manipulated, wrong object state or placement, wrong order, missing subtask, and contradictory subtasks. Execution failures are generated online in simulation by perturbing subtask-level actions (no gripper close, wrong object state/placement, wrong object manipulated, imprecise grasping/pushing), and offline on real-robot data by perturbing the paired subtask instruction (task-execution semantic mismatch) or reverting the action to show no progress.

The resulting dataset, GuardianFail-36k, consists of RLBench-Fail and BridgeDataV2-Fail, with over 36K examples. It contains balanced success/failure samples for both planning and execution, multi-view observations, and explicit chain-of-thought supervision. Our designed failure modes reflect the failure distribution observed when running a real policy (3D-LOTUS++) on RLBench.



Chain-of-Thought (CoT) Generation. To improve reasoning and interpretability, we automatically generate structured chain-of-thought (CoT) annotations for each example. For every sample, we extract: Object categories and spatial information Robot state and task context Ground-truth failure reasons We then prompt a large reasoning-capable VLM (InternVL3-38B) to produce step-by-step reasoning traces (118 tokens on average). For planning, the model verifies each subtask sequentially and evaluates overall plan correctness For execution, it compares pre- and post-action observations before assessing success These reasoning traces provide fine-grained supervision and significantly improve failure detection performance, while enabling interpretable decision-making.



Real-Robot, Policy-Driven Data Collection. To further support realistic evaluation, we curate UR5-Fail, a new real-world dataset collected using a UR5 arm with three cameras. We run the 3D-LOTUS++ policy on 16 unique tasks and record initial and final multi-view images for each subtask. Execution outcomes are manually labeled as success or failure. Planning failures are generated using the same perturbation framework described above. Compared to prior datasets, UR5-Fail: Uses multi-view observations instead of single-view Includes autonomous policy rollouts instead of teleoperation Captures more realistic and diverse failure modes


Method: The Guardian Model



Model Architecture and Integration into a Robotic Manipulation Framework. Left: Overview of the Guardian model architecture. Right: Integration of Guardian model into a robot manipulation pipeline for planning and execution verification.

Guardian is a multi-view reasoning VLM designed for unified failure detection at both planning and execution stages. It is built upon the open-source InternVL3-8B (InternViT-300M visual encoder + Qwen2.5-7B LLM) and fine-tuned on GuardianFail-36k with LoRA, keeping the visual encoder frozen. Unlike prior methods that concatenate images into a single grid image, Guardian: Processes each view independently, preserving spatial detail Explicitly models temporal changes between observations Generates an explicit step-by-step reasoning trace before predicting success or failure We formulate failure detection as a visual question answering problem: Planning verification: assess whether a proposed plan is valid given the task and initial observation Execution verification: determine whether a subtask was successfully completed based on before/after observations We explore three CoT strategies (Vanilla, Thinking, Dropout): training with reasoning traces consistently improves accuracy, with Thinking performing best and Dropout offering a better speed-accuracy trade-off. Guardian can be seamlessly plugged into existing robotic manipulation pipelines as a verification layer, without any architectural modification, enabling: i) detection of failures at each planning and subtask execution step, ii) triggering of replanning or re-execution of the motion policy, iii) use of the fine-grained failure reasoning as a hint to replan


Quantitative results on Robotic Manipulation Failure Detection Benchmarks



Guardian achieves the strongest overall performance across three unseen real-world benchmarks, RoboFail, UR5-Fail and RoboVQA, compared to GPT-4o and specialized robotic failure detection VLMs (RoboFAC, AHA, I-Fail-Sense, Cosmos-Reason). Training on GuardianFail-36k consistently improves all architectures, and combining simulated (RLBench-Fail) and real (BridgeDataV2-Fail) data yields consistent gains across benchmarks, with performance scaling steadily with dataset size.

Qualitative examples in simulation



Integrated as a verifier into 3D-LOTUS++ on 10 unseen RLBench tasks, Guardian improves the average success rate from 0.45 (no verifier) to 0.54, with the failure reasoning fed back to support replanning. On a real UR5 robot (three unseen tasks, nominal and perturbed settings), Guardian also brings consistent improvements, especially under perturbations.

Qualitative examples in the real world



Left and middle: successful planning and execution verification on UR5-Fail; right: a failure case where Guardian misjudges the grasp as successful. Scaling structured failure reasoning data—across simulation and real-world domains—is critical to building robust robotic systems. By combining automated failure generation with multi-view reasoning, Guardian enables reliable detection and correction of both planning and execution errors.

BibTeX


@article{pacaud2026guardian,
      title={Guardian: Detecting Robotic Manipulation Failures with Vision-Language Models},
      author={Paul Pacaud and Ricardo Garcia Pinel and Shizhe Chen and Cordelia Schmid},
      year={2026},
      journal={arXiv preprint arXiv:2512.01946}
}

Acknowledgements

This work was performed using HPC resources from GENCI-IDRIS (Grant 2025-AD011015795 and AD011015795R1). It was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “France 2030" program, reference ANR-23-IACL-0008 (PR[AI]RIE-PSAI project), the ANR project VideoPredict ANR-21-FAI1-0002- 01. Cordelia Schmid would like to acknowledge the support by the Körber European Science Prize.