Extrapolative Weight Averaging Reveals Correctness-Efficiency Frontiers in Code RL

Researchers demonstrate that extrapolative weight averaging—extending beyond trained model checkpoints—can navigate and extend correctness-efficiency frontiers in code reinforcement learning without additional training. Testing on competitive programming tasks reveals that ensembles using this technique improve performance by 3.3% on hard problems, suggesting a scalable method for optimizing AI systems across competing objectives.

This research addresses a fundamental challenge in AI optimization: balancing multiple competing objectives without expensive retraining cycles. The study explores how extrapolative weight averaging can extend beyond the Pareto frontiers established by linear interpolation between fine-tuned checkpoints, creating new inference-time solutions that weren't explicitly trained. In the context of code generation and competitive programming, this manifests as a tension between functional correctness and computational efficiency—solving problems correctly but within strict time and memory constraints.

The work builds on established findings that model checkpoint interpolation traces Pareto fronts, extending this principle into unexplored territory. By training models under nested unit-test coverage regimes, researchers engineered a controlled sweep where different training objectives naturally produced checkpoints at different points along a correctness-efficiency frontier. The key discovery is that extrapolation beyond these endpoints yields useful new checkpoints without requiring additional RL training cycles, dramatically reducing computational cost.

For AI practitioners and organizations deploying code generation systems, this offers immediate practical value. The technique demonstrates that ensemble methods combining extrapolated checkpoints improve overall performance on hard problems, with measurable gains in pass rates. The method generalizes across different model scales (32B and 7B parameters) and inference paradigms including pure reasoning, tool use, and agentic coding, suggesting broad applicability.

The implications extend to how organizations approach multi-objective optimization in production AI systems. Rather than training separate models for each objective or accepting compromises, teams can now leverage weight averaging as an inference-time scaling technique. The emergence of complementary policies that solve different problem subsets enables more efficient ensemble strategies, potentially reducing infrastructure costs while improving performance.

• →Extrapolative weight averaging extends correctness-efficiency frontiers in code RL without additional training, reducing computational overhead
• →Nested unit-test coverage during training naturally produces checkpoints distributed along Pareto frontiers that enable effective extrapolation
• →Ensembles combining extrapolated checkpoints improve hard problem performance by 3.3% compared to best single checkpoint at equivalent sample budgets
• →The technique generalizes across model scales and inference settings, from pure reasoning to agentic coding systems
• →Extrapolated checkpoints act as complementary policies solving different problem subsets, enabling efficient inference-time optimization