V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions

Researchers introduce V-REX, a new evaluation benchmark for vision-language models that assesses their ability to perform complex, multi-step visual reasoning through Chain-of-Questions (CoQ) methodology. The framework disentangles VLMs' planning and information-gathering capabilities, revealing significant performance gaps and substantial room for improvement in exploratory visual reasoning tasks.

V-REX addresses a critical gap in how vision-language models are evaluated. While current benchmarks focus on straightforward question-answering with predefined targets, real-world applications demand iterative exploration and reasoning across visual data. This research introduces a structured methodology to measure these capabilities systematically, moving beyond single-turn interactions to multi-step problem-solving.

The framework's innovation lies in its Chain-of-Questions approach, which breaks down complex visual reasoning into two distinct competencies: planning (formulating exploratory questions) and following (executing curated question sequences). By constraining intermediate steps to finite options, V-REX enables quantitative, fine-grained analysis previously impossible with open-ended exploration spaces. This represents a methodological advancement in AI evaluation alongside practical improvements in understanding model capabilities.

For the AI development community, V-REX's findings carry significant implications. Testing both proprietary and open-source models reveals consistent scaling trends and highlights substantial performance gaps between planning and following abilities. This diagnostic insight enables developers to target specific weaknesses rather than broadly improving model architectures. The benchmark spans diverse application domains, suggesting findings generalize across use cases rather than representing narrow optimizations.

Looking forward, V-REX establishes a foundation for iterative improvement in visual reasoning capabilities. As VLMs increasingly handle complex, real-world tasks requiring exploration and verification, benchmarks like this become essential for progress tracking. The framework may influence how subsequent model generations are evaluated and trained, potentially reshaping development priorities toward multi-step reasoning rather than isolated task performance.

• →V-REX introduces Chain-of-Questions methodology to evaluate multi-step exploratory visual reasoning in VLMs, addressing gaps in existing benchmarks
• →The framework disentangles planning (question formulation) and following (sequential answering) abilities, revealing significant performance differences between these capabilities
• →Current state-of-the-art models show substantial room for improvement in complex, open-ended visual reasoning tasks requiring native multi-step exploration
• →Finite-option curated steps enable reliable quantitative analysis of intermediate reasoning processes previously difficult to evaluate systematically
• →Consistent scaling trends across diverse domains suggest findings apply broadly to VLM development rather than specific use cases