EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents

Researchers introduce EgoBench, a new benchmark for evaluating AI agents' ability to perceive visual information, reason through multi-step tasks, and interact with users in real-world scenarios. Testing eight state-of-the-art video models reveals significant limitations, with the best performer achieving only 30.62% accuracy, exposing critical gaps in current AI agent capabilities.

EgoBench addresses a fundamental evaluation gap in AI agent development. As autonomous systems move toward real-world deployment, they must simultaneously process visual information, invoke tools through complex reasoning chains, and respond to dynamic user feedback. Existing benchmarks measure these capabilities in isolation, failing to capture the integrated performance required for practical applications. The introduction of 1,045 egocentric video tasks across four daily scenarios, paired with a simulated user agent that generates contextually appropriate feedback, creates a more realistic testing environment.

The benchmark's three-stage pipeline deliberately couples visual perception with multi-hop reasoning, forcing models to demonstrate genuine understanding rather than pattern matching. The multi-agent simulated user component represents a significant methodological advancement, enabling evaluation of interaction quality that standard static benchmarks cannot assess. The deterministic validation framework ensures reproducibility and objective scoring through both process-based and result-based equivalence metrics.

Performance results carry substantial implications for the AI agent market. The 30.62% ceiling on best-case scenarios and 19.43% average accuracy across scenarios reveals that current video-multimodal language models remain far from production-ready for autonomous task execution. This performance gap directly impacts development timelines for enterprise AI agent adoption, suggesting significant engineering work remains before deployment in safety-critical applications. The error analysis identifying specific capability bottlenecks provides developers actionable targets for improvement, potentially accelerating the next generation of AI agent design.

• →EgoBench is the first benchmark simultaneously evaluating multimodal perception, tool-use reasoning, and user interaction for AI agents.
• →Top-performing models achieve only 30.62% accuracy on the benchmark's best scenario, indicating substantial gaps in current AI agent capabilities.
• →The benchmark introduces a simulated user agent that generates task-aligned feedback, enabling dynamic interaction evaluation previously unavailable in static benchmarks.
• →Results suggest current video-MLLM agents require significant improvements before deployment in real-world autonomous task execution scenarios.
• →Detailed error analysis identifies specific failure modes and capability bottlenecks for guiding future AI agent development.