OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics

Researchers introduce OmniGameArena, a comprehensive UE5-based benchmark for evaluating vision-language model agents across diverse game environments (solo, PvP, cooperative), along with the Improvement Dynamics Curve methodology that tracks agent performance evolution through iterative refinement rather than single snapshots.

OmniGameArena addresses a critical gap in VLM agent evaluation by moving beyond one-dimensional performance metrics. Traditional game benchmarks capture only initial success rates, providing limited insight into agent capabilities or learning potential. This new framework introduces temporal measurement through the Improvement Dynamics Curve, which tracks how agents refine their strategies across multiple reflection rounds using tool-augmented LLMs. The benchmark's inclusion of twelve heterogeneous games spanning solo, competitive, and cooperative modes reflects real-world deployment diversity that single-game evaluations cannot capture.

The research builds on growing momentum in embodied AI evaluation, where static benchmarks have proven insufficient for understanding agent adaptability. By incorporating game variants and tracking skill transfer, OmniGameArena generates richer behavioral signals about agent robustness and generalization capabilities. The unified action interface framework enables fair comparison between commercial VLMs (Claude, GPT-4V), open-weight alternatives, and specialized policies—categories previously evaluated in isolation.

This work carries implications for the AI development ecosystem. Better evaluation mechanisms reduce information asymmetry between model creators and users, enabling more informed deployment decisions. For open-source communities, unified benchmarking standards accelerate competitive advancement by establishing clear performance baselines. The reflection-based improvement dynamics expose whether agents genuinely learn strategic insights or merely exploit initial prompt engineering, revealing qualitative differences between systems claiming similar performance levels.

Future attention should focus on whether industry adoption standardizes around OmniGameArena's protocol and whether improvement curves correlate with downstream task performance in real applications. Extended evaluation on procedurally generated game variants could further stress-test generalization claims.

• →OmniGameArena moves beyond single-attempt scores by tracking agent performance evolution across multiple reflection rounds using tool-using LLMs
• →The benchmark spans 12 Unreal Engine 5 games across solo, PvP, and cooperative modes with unified action interfaces for fair cross-agent comparison
• →Improvement Dynamics Curves expose how agents refine strategies iteratively and transfer learned skills to held-out game variants
• →Framework enables standardized evaluation of heterogeneous agent classes including commercial VLMs, open-weight models, and specialized game policies
• →Richer behavioral signals help distinguish genuine agent learning from prompt engineering artifacts in performance comparisons