OpenWebRL: Demystifying Online Multi-turn Reinforcement Learning for Visual Web Agents

Researchers introduce OpenWebRL, an open-source framework for training visual web agents using online reinforcement learning directly on live websites. The resulting OpenWebRL-4B model achieves state-of-the-art performance on web-based benchmarks with minimal training data, challenging the proprietary-system dominance and offering a scalable alternative to expensive supervised learning approaches.

OpenWebRL addresses a critical limitation in open-source AI development: the scalability bottleneck created by dependence on curated demonstration datasets. Traditional approaches to training visual web agents require expensive human-annotated trajectories, limiting coverage of the diverse and constantly-evolving web. This research demonstrates that online reinforcement learning—previously underexplored for visual agents—can train capable systems directly on live websites, fundamentally changing the economics of agent development.

The framework achieves remarkable efficiency, reaching competitive performance with only 400 initialization trajectories and 2,200 RL training tasks. OpenWebRL-4B's 67% success rate on Online-Mind2Web and 64% on DeepShop benchmarks rivals or exceeds larger proprietary systems from OpenAI and Google, suggesting that algorithmic innovation matters more than scale alone. The paper's systematic analysis of design choices provides actionable insights for the broader research community.

For the AI industry, this work democratizes visual agent development and reduces barriers to entry. Smaller organizations and academic teams can now compete with well-resourced labs without massive labeled datasets. The commitment to release training data, models, and code amplifies impact by enabling reproducible research and accelerating downstream innovation.

Looking forward, online RL for web agents likely becomes a standard training paradigm. The approach's efficiency suggests that future improvements in reasoning, grounding, and real-world interaction will compound rapidly. This shift from static datasets to live-environment learning may reshape how AI systems interact with dynamic digital ecosystems, with implications extending beyond web browsing to broader autonomous agent applications.

• →OpenWebRL achieves state-of-the-art open-source performance with 67% success on live-web benchmarks using minimal training data.
• →Online RL dramatically improves scalability by training directly on live websites rather than requiring expensive curated demonstrations.
• →OpenWebRL-4B matches or exceeds proprietary systems from OpenAI and Google despite similar or smaller model size.
• →The framework's systematic design analysis provides actionable insights for training visual agents effectively.
• →Releasing code, models, and data will democratize visual agent development and accelerate community research.