EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning

Researchers introduce EvoTrainer, an autonomous framework that co-evolves large language model policies and training harnesses through empirical feedback, matching or exceeding human-engineered reinforcement learning baselines across mathematical reasoning, code generation, and software engineering tasks. The approach moves beyond static recipe-based training to jointly optimize both policies and the training infrastructure that interprets them.

EvoTrainer addresses a fundamental limitation in autonomous LLM training: the assumption that training harnesses should remain static while only policies evolve. Traditional recipe-search approaches to autonomous training fail to capture the dynamic nature of reinforcement learning in agentic systems, where performance bottlenecks shift and scalar rewards obscure diverse failure modes. This research demonstrates that co-evolving both the policies and the training infrastructure yields measurably better results.

The framework operates through a diagnostic feedback loop, where it analyzes rollout-level evidence, refines diagnostic approaches, backtests proposed interventions, and accumulates reusable skills across training runs. Testing on three demanding domains—mathematical reasoning, competitive programming, and repository-level software engineering—reveals that EvoTrainer matches or exceeds hand-tuned baselines, with particularly strong gains in long-horizon agentic software engineering tasks where complexity compounds.

Trajectory analysis reveals domain-specific strategy divergence, indicating the system adapts to unique problem structures rather than converging on generic solutions. The prevention of invalid high-scoring branches through evolving diagnostics suggests the framework develops robustness against spurious reward signals. The accumulation of reusable skills across search phases demonstrates emergent knowledge transfer.

This work has implications for the broader AI training paradigm. As LLM systems become increasingly agentic and long-horizon tasks dominate, static training harnesses become genuine bottlenecks. The research validates that infrastructure optimization deserves equal consideration to policy optimization, potentially reshaping how organizations approach autonomous AI development and reducing dependency on human expert intuition in training design.

• →EvoTrainer co-evolves LLM policies and training harnesses autonomously, outperforming static human-engineered RL baselines on multiple complex domains.
• →The framework diagnoses failures at the rollout level, revises diagnostics iteratively, and accumulates reusable skills across training cycles.
• →Domain-specific strategy divergence shows the system adapts to problem structure rather than learning generic solutions.
• →Evolving diagnostics prevent invalid high-scoring branches from being promoted, improving robustness against spurious reward signals.
• →Results suggest autonomous LLM training should jointly optimize both policies and training infrastructure, not just policies alone.