Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents

Researchers introduce VIGIL, an evaluation framework that separately measures whether embodied AI agents correctly complete tasks and properly report success, rather than conflating execution failures with commitment failures. Testing across 20 models reveals significant performance gaps in terminal commitment despite similar task execution, highlighting a critical blind spot in current AI agent benchmarking.

The paper addresses a fundamental measurement problem in embodied AI evaluation: existing benchmarks cannot distinguish between agents that fail to complete tasks, agents that complete tasks but fail to stop, and agents that claim success without evidence. VIGIL decouples world-state completion (W) from benchmark success (B), making terminal commitment independently visible. This matters because behavioral failure modes that look identical in current metrics actually reflect distinct underlying deficiencies.

The research emerges from growing sophistication in embodied AI systems, where agents operate in simulated or real environments requiring sequential decision-making. As these systems become more capable, crude pass-fail metrics obscure their actual reliability. An agent that achieves a goal but drifts past it without reporting completion presents different safety and usability challenges than one that never attempts the goal—yet standard benchmarks treat both as failures.

For AI developers and researchers, VIGIL's findings carry immediate implications. The 19.7 percentage-point gap between models with similar execution performance but different commitment success rates suggests that terminal commitment is learnable but not automatically acquired through execution training. The action-feedback intervention demonstrates that execution-oriented signals improve task completion broadly, yet commitment failures persist in models lacking grounding between achieved states and terminal reports. This indicates agents need explicit training to reliably report their own success.

Moving forward, adoption of VIGIL-style evaluation protocols could reshape how embodied AI systems are trained and selected. The framework particularly matters for real-world deployment scenarios where an agent silently succeeding differs critically from one that communicates success reliably.

• →VIGIL separates task execution from terminal commitment, revealing up to 19.7pp performance gaps invisible to standard metrics
• →Four distinct failure modes—missed execution, post-attainment drift, unsupported commitment, and verified success—become distinguishable under the framework
• →Execution-improving interventions do not automatically improve commitment, requiring explicit grounding between achieved states and terminal reports
• →Current embodied AI benchmarks conflate behaviorally distinct failures, obscuring critical safety and reliability differences
• →Terminal commitment represents a learnable but underexplored capacity in deployed embodied agents