Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring

Researchers propose Hide-and-Seek, a machine learning framework that detects failures in Vision-Language-Action (VLA) models during robot execution by identifying failure-indicative actions from trajectory-level data alone. The method achieves state-of-the-art performance across multiple VLA policies and robotic platforms without requiring expensive step-level annotations or external models.

Hide-and-Seek addresses a critical reliability challenge in embodied AI systems where Vision-Language-Action models must safely execute complex robotic tasks. Current failure detection approaches either require computationally expensive action resampling, external monitoring systems, or trajectory-level labels that fail to pinpoint when and why failures occur during execution. This research matters because deploying VLA models in real-world robotics demands robust failure detection to ensure safety and reliability.

The framework reformulates failure detection as a coarsely supervised learning problem, combining inter-trajectory and intra-trajectory contrastive learning objectives. This dual approach enables the model to identify failure-indicative actions and induce temporally structured failure signals using only trajectory-level labels, eliminating the need for detailed step-by-step annotations. The method demonstrates strong empirical results across LIBERO and VLABench benchmarks, plus real robotic deployments, testing against OpenVLA, π₀, and π₀.₅ policies.

For the AI development community, this work reduces annotation overhead while improving failure detection granularity—a practical advantage for scaling robotic systems. The generalization to unseen tasks suggests the approach captures fundamental failure patterns rather than memorizing specific scenarios. The accuracy-timeliness trade-off under conformal prediction indicates practitioners can configure detection thresholds based on deployment requirements.

Looking forward, integration of such failure detection mechanisms into production VLA systems will likely accelerate real-world robotic adoption. Future research may explore transfer learning across different robot morphologies and extending temporal reasoning to multi-step failure cascades.

• →Hide-and-Seek detects VLA execution failures using only trajectory-level labels, eliminating expensive step-level annotation requirements
• →The method combines inter-trajectory and intra-trajectory contrastive learning to localize failure-indicative actions with temporal structure
• →Framework achieves state-of-the-art performance across three major VLA policies and generalizes effectively to unseen robotic tasks
• →Conformal prediction enables practical accuracy-timeliness trade-offs for configurable failure detection thresholds in deployment
• →Real-world robotic validation demonstrates the approach's applicability beyond simulation benchmarks