MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation

Researchers introduce MCTS-Judge, a test-time scaling framework that enhances LLM-based code evaluation by applying Monte Carlo Tree Search to improve reasoning accuracy. The system achieves 80% accuracy on code correctness tasks—surpassing OpenAI's o1 models while using 3x fewer tokens—addressing a critical limitation in using LLMs as reliable judges for complex technical problems.

MCTS-Judge represents a meaningful advance in test-time computation applied to code evaluation, a domain where LLM reasoning has historically struggled. The framework tackles a genuine problem: standard LLM-as-a-Judge approaches lack the systematic reasoning depth required for programming tasks that demand line-by-line logical verification. By combining Monte Carlo Tree Search with refined reward mechanisms, the system decomposes complex code analysis into manageable evaluations from multiple perspectives, enabling the base model to reason more rigorously.

This work builds on broader momentum in scaling test-time computation rather than model size alone. Recent reasoning models like OpenAI's o1 demonstrated that allocating computation at inference time yields better results than simply increasing model parameters. MCTS-Judge takes this insight and applies it specifically to the evaluation domain, where reliability directly impacts downstream development and deployment decisions.

The efficiency gains carry practical significance: achieving o1-level accuracy with 3x fewer tokens reduces computational costs substantially, making sophisticated code evaluation accessible to organizations with constrained infrastructure budgets. For developers and ML teams, this suggests that rigorous automated code review may become more economically viable without requiring expensive frontier model access.

Moving forward, the key question is adoption velocity. If MCTS-Judge generalizes effectively across diverse code types and programming paradigms, it could reshape how organizations validate generated code. The research validates the broader trend that test-time scaling deserves equal investment alongside model scaling, potentially influencing how future reasoning-intensive evaluation systems are designed.

• →MCTS-Judge achieves 80% accuracy on code evaluation benchmarks, improving base model performance by 39 percentage points
• →The framework matches o1-series model quality while consuming 3x fewer tokens, significantly reducing computational costs
• →Monte Carlo Tree Search decomposition enables multi-perspective code analysis with unit-test-level precision
• →Test-time computation scaling demonstrates measurable benefits for reasoning-intensive evaluation tasks beyond generation
• →Efficiency gains make sophisticated LLM-based code review economically viable for resource-constrained development teams