Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge

Researchers demonstrate that reasoning-capable LLMs improve judgment accuracy significantly on complex tasks like math and coding, but offer minimal or negative benefits on simpler evaluations while consuming substantially more computational resources. They introduce RACER, an adaptive routing algorithm that dynamically selects between reasoning and non-reasoning judges under budget constraints while accounting for distribution shifts.

The research addresses a critical inefficiency in deploying large language models as automated judges. While reasoning-enhanced models like OpenAI's o1 have shown promise in evaluation tasks, the study reveals that their universal application represents poor resource allocation. The controlled comparison demonstrates clear task-dependent returns: structured verification problems benefit substantially from explicit reasoning chains, whereas simpler judgments waste computational budget without meaningful accuracy gains.

This findings emerge from growing adoption of LLMs in evaluation pipelines across academia and industry. Companies increasingly use LLM judges to evaluate model outputs, content moderation, and quality assurance, but the cost-benefit analysis has remained opaque. The paper's contribution moves beyond anecdotal observations to empirical quantification of when reasoning actually matters.

RACER's technical approach addresses real deployment constraints. By formulating routing as a distributionally robust optimization problem with KL-divergence uncertainty sets, the algorithm protects against performance degradation when task distributions shift—a practical concern when evaluation criteria evolve. The primal-dual algorithm provides computational efficiency with convergence guarantees, making it implementable at scale.

For developers and platforms deploying LLM judges, this research carries direct implications. Rather than defaulting to expensive reasoning models, selective routing based on task characteristics can reduce inference costs by orders of magnitude while maintaining accuracy. This optimization becomes increasingly important as evaluation volumes scale. The work also signals that AI practitioners should rigorously benchmark tool choices rather than assuming frontier models universally outperform alternatives.

• →Reasoning LLMs significantly improve judgment accuracy only on structured verification tasks like math and coding problems, not simpler evaluations
• →RACER algorithm dynamically routes between reasoning and standard judges to optimize accuracy-cost trade-offs within fixed computational budgets
• →Distribution shift awareness is critical—the algorithm uses KL-divergence uncertainty sets to maintain performance as task distributions evolve
• →Selective reasoning deployment can substantially reduce inference costs compared to universal application of expensive reasoning models
• →Theoretical guarantees including optimal policy uniqueness and linear convergence make RACER practically implementable at scale