ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Models

Researchers introduce ViPlan, the first benchmark for comparing Vision-Language Model planning approaches, finding that VLM-as-grounder methods excel in visual tasks like Blocksworld while VLM-as-planner methods perform better in household robotics scenarios. The study reveals fundamental limitations in current VLMs' visual reasoning abilities, with Chain-of-Thought prompting showing no consistent benefits.

• →ViPlan is the first open-source benchmark to compare VLM-grounded symbolic planning with direct VLM planning methods.
• →VLM-as-grounder approaches solved 46% of Blocksworld tasks compared to 9% for direct VLM planning methods.
• →In household robotics tasks, VLM-as-planner methods significantly outperformed VLM-as-grounder approaches (34% vs 5% success rate).
• →Chain-of-Thought prompting showed no consistent benefits across methods, highlighting persistent VLM limitations.
• →The benchmark reveals that different planning approaches excel in different domains based on visual complexity and linguistic knowledge requirements.