CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences

Researchers introduce CV-Arena, a benchmark containing 12,000 high-resolution image instruction pairs to evaluate how well AI systems solve professional-grade computer vision tasks. The study proposes Active Elo, a human-AI collaborative evaluation protocol, and reveals that current models struggle with instruction adherence, physical reasoning, and detail preservation in real-world editing workflows.

CV-Arena represents a significant advancement in how the computer vision community measures AI capabilities for practical image editing tasks. Rather than focusing on narrow appearance modifications, this benchmark mirrors professional workflows by testing systems against constraints involving preservation, geometry, physics, and usability—dimensions that simple benchmarks have historically overlooked. The research team's dual-track construction methodology combines web retrieval with agentic query refinement, ensuring the dataset captures genuine professional demands rather than synthetic or overly simplified scenarios.

The Active Elo evaluation protocol addresses a critical challenge in AI assessment: scaling human judgment without sacrificing accuracy. By using CV-Judge, a logic-gated vision-language model, to filter obvious failures and high-confidence cases before routing ambiguous comparisons to expert raters, the framework maintains human fidelity while reducing annotation costs. This hybrid approach reflects maturation in AI evaluation methodologies, acknowledging that purely automated metrics often miss nuanced quality differences.

The evaluation of 21 systems reveals systematic weaknesses across current approaches, particularly in instruction adherence and structural control. These gaps suggest that instruction-following visual editing remains an unsolved problem despite recent advances in multimodal AI. The introduction of CV-Agent, a lightweight agentic system that iterates through planning, editing, and verification cycles, demonstrates that closed-loop reasoning can address some limitations, pointing toward multi-turn problem-solving as the next frontier. For developers building professional visual tools, this research provides both a rigorous evaluation standard and evidence that single-pass editing models are insufficient for enterprise workflows.

• →CV-Arena provides a 12K high-resolution benchmark capturing 16 real-world instruction-based vision tasks, bridging the gap between academic benchmarks and professional image editing requirements.
• →Active Elo introduces a hybrid human-AI evaluation protocol that scales assessment while preserving expert judgment, addressing efficiency challenges in large-scale model evaluation.
• →Current AI systems, including proprietary and open-source models, consistently fail at instruction adherence, physical reasoning, and fine-grained detail preservation in complex editing scenarios.
• →Agentic approaches with closed-loop reasoning and multi-turn verification outperform single-pass models, suggesting a shift toward iterative problem-solving for professional-grade visual editing.
• →The research establishes a new standard for benchmarking visual instruction-following systems that accounts for real-world constraints beyond appearance matching.