CodeGolf Bench: A Multi-Language Benchmark for Evaluating Concise Code Generation Capabilities of Large Language Models

Researchers introduce CodeGolf Bench, a new benchmark for evaluating Large Language Models' ability to generate concise code across 60 programming languages. The study reveals that reasoning-capable models significantly outperform standard LLMs, achieving 70.97% average percentile performance on code golf tasks, particularly excelling in languages with strict syntax requirements.

CodeGolf Bench addresses a meaningful gap in LLM evaluation frameworks by introducing a dynamic benchmarking approach grounded in actual competitive programming. Code golf represents a specialized but revealing test of model capabilities—producing minimal, efficient code requires both algorithmic understanding and language-specific optimization knowledge that differs substantially from standard code generation tasks. The benchmark's connection to the code.golf platform provides continuous problem rotation and human performance baselines, preventing the benchmark stagnation that affects many existing evaluation frameworks.

The performance divergence between reasoning and non-reasoning models carries significant implications for understanding current AI capabilities. Reasoning models achieving 70.97% percentile performance suggests these systems can optimize beyond surface-level solutions, tackling the constraint-satisfaction problem inherent to code golf. The pronounced gap in C++ performance indicates that strict syntax languages demand the kind of step-by-step logical processing that reasoning architectures provide. Standard models' struggles with efficiency optimization point to a fundamental limitation in how non-reasoning models approach multi-objective problems requiring both correctness and conciseness.

For developers and AI practitioners, this benchmark provides actionable intelligence about model selection for production code generation tasks. Organizations requiring efficiency-optimized code should prioritize reasoning models, particularly for typed languages. The dynamic framework suggests future benchmarks may increasingly move toward live, evolving evaluation systems rather than static problem sets. This research also illuminates where current non-reasoning models face genuine capability boundaries, informing architecture improvements and training methodology development for next-generation systems.

• →Reasoning-capable LLMs achieve 70.97% average percentile on code golf tasks, vastly outperforming non-reasoning models
• →Performance gaps are most pronounced in strict-syntax languages like C++, highlighting reasoning's importance for syntactic complexity
• →CodeGolf Bench provides dynamic, evolving benchmarking against live human performance rather than static problem sets
• →Non-reasoning models struggle significantly with code efficiency optimization, revealing fundamental architectural limitations
• →The benchmark covers 60 programming languages, providing the broadest language coverage for conciseness-focused LLM evaluation