ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis

Researchers present ScaleWoB, a framework that synthesizes high-fidelity interactive environments for training and evaluating GUI agents across mobile, desktop, and automotive platforms. The approach addresses critical limitations of real-world testing by providing verifiable rewards, low resource costs, and accessibility via URL-based backends, with results showing state-of-the-art agents achieve only 27.92% success compared to 92.08% for humans.

ScaleWoB addresses a fundamental challenge in AI agent development: the difficulty of creating scalable, reproducible testing environments for GUI-based systems. Traditional approaches rely on real-world environments that are complex, uncontrollable, and resource-intensive, or on limited virtual setups that don't reflect authentic usage patterns. This framework solves these constraints by generating synthetic but realistic interactive environments that function as backend-free webpages, dramatically reducing infrastructure overhead while maintaining fidelity.

The broader context reflects an accelerating trend in AI evaluation infrastructure. As large language models become increasingly capable at complex task execution, the bottleneck shifts from model capability to environment quality and evaluation rigor. Previous GUI agent benchmarks were limited to open-source applications or file operations, creating a persistent gap between benchmark performance and real-world capability. ScaleWoB's support for 100+ environments and 1000+ verifiable tasks across multiple platforms represents a significant expansion in scope and realism.

The performance gap revealed in the research—27.92% average success for state-of-the-art agents versus 92.08% for humans—demonstrates substantial headroom for improvement and validates the benchmark's difficulty level. The fact that assessments generalize from synthetic to real applications provides confidence in the evaluation methodology, suggesting these benchmarks predict genuine real-world capability rather than gaming specific datasets.

For the AI industry, this infrastructure enables both more rigorous agent evaluation and efficient large-scale training. The low resource requirements make ScaleWoB accessible to researchers with limited computational budgets, potentially democratizing advanced agent development. Future developments should focus on expanding task diversity, increasing environmental complexity, and exploring how synthetic training translates to real-world deployment effectiveness.

• →ScaleWoB synthesizes 100+ GUI environments with 1000+ verifiable tasks across mobile, desktop, and automotive platforms using zero-setup backend-free architecture
• →State-of-the-art GUI agents achieve only 27.92% average success versus 92.08% for humans, revealing substantial capability gaps and benchmark difficulty
• →Framework reduces resource requirements and evaluation costs by eliminating virtual machines and Docker dependencies while maintaining reproducibility
• →Synthetic environment assessments generalize to real-world applications, validating the benchmark's relevance for practical agent evaluation
• →Low-cost, scalable infrastructure enables both large-scale agent evaluation and downstream training for resource-constrained research teams