SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios

Researchers introduce SWITCH, a new benchmark for testing autonomous AI agents' ability to interact with physical interfaces like switches and appliance panels in real-world scenarios. The benchmark reveals significant gaps in current AI models' capabilities for long-horizon tasks requiring causal reasoning and verification.

• →SWITCH benchmark evaluates AI agents on five key abilities including task-aware VQA, semantic UI grounding, and action generation across 351 tasks.
• →Testing covers 98 real devices and appliances to assess agents' interaction with tangible control interfaces in everyday environments.
• →Commercial and open-source large multimodal models showed systematic failures in handling long-horizon embodied scenarios.
• →The benchmark addresses critical gaps in partial observability, causal reasoning across time, and failure-aware verification.
• →Resources are publicly available to enable reproducible evaluation and community contributions for future iterations.