Beyond Scaling: Assessing Strategic Reasoning and Rapid Decision-Making Capability of LLMs in Zero-sum Environments

Researchers introduce STAR Benchmark, a new evaluation framework for testing Large Language Models in competitive, real-time environments. The study reveals a strategy-execution gap where reasoning-heavy models excel in turn-based settings but struggle in real-time scenarios due to inference latency.

• →STAR Benchmark introduces multi-agent competitive evaluation for LLMs in zero-sum environments.
• →Current LLM evaluations fail to assess opponent-aware decision-making and temporal constraints.
• →Reasoning-intensive models dominate turn-based strategic games but underperform in real-time settings.
• →Faster instruction-tuned models show superior performance in time-sensitive competitive scenarios.
• →Strategic intelligence requires both reasoning depth and ability to execute timely actions.