LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

Researchers introduce LLM-WikiRace, a benchmark that tests large language models' planning and reasoning abilities by requiring them to navigate Wikipedia links from a source to target page. While frontier models like Gemini-3 achieve superhuman performance on easy tasks, success rates plummet to 23% on hard difficulty, revealing significant limitations in long-horizon planning and recovery from failures.

LLM-WikiRace addresses a critical gap in AI evaluation by moving beyond traditional benchmarks to test real-world planning and reasoning over knowledge graphs. The benchmark reveals a nuanced picture of current LLM capabilities: while these models excel at leveraging world knowledge on easier tasks, this advantage diminishes sharply as complexity increases, suggesting that memorization of training data has hit an effective ceiling.

The sharp performance degradation on hard tasks—where the best model succeeds only 23% of the time—exposes a fundamental weakness in current reasoning systems. The research demonstrates that planning and long-horizon reasoning, rather than raw knowledge, become the bottleneck for solving complex multi-step problems. This aligns with broader observations in AI research showing that scaling model size and training data yields diminishing returns on complex reasoning tasks.

The finding that models frequently enter loops and fail to replan after mistakes has significant implications for real-world applications. Systems deployed in navigation, scientific discovery, or decision-support roles would encounter similar failure modes when confronted with unfamiliar scenarios requiring adaptive strategies. This limitation suggests current LLMs remain far from achieving robust general-purpose planning agents.

The open-source leaderboard approach creates a competitive research ecosystem that could accelerate improvements in planning capabilities. Developers and AI companies face pressure to demonstrate progress on this benchmark as evidence of advancing reasoning abilities. The test establishes concrete performance targets that distinguish between marketing claims and genuine capability improvements, making it increasingly difficult for vendors to claim superhuman reasoning without empirical validation on challenging tasks.

• →Frontier LLMs succeed only 23% of the time on hard WikiRace tasks despite superhuman easy-level performance, revealing planning as the key limiting factor
• →World knowledge becomes less important than reasoning ability for complex multi-step problems, challenging the scaling paradigm that dominates current AI development
• →Models consistently fail to replan after mistakes and frequently enter loops, indicating poor recovery mechanisms for real-world deployment scenarios
• →The benchmark's open leaderboard creates measurable standards for evaluating planning capabilities beyond traditional knowledge-based metrics
• →Success on this benchmark requires a fundamental shift beyond knowledge retrieval toward robust long-horizon reasoning and adaptive problem-solving