Evolving-RL: End-to-End Optimization of Experience-Driven Self-Evolving Capability within Agents

Researchers introduce Evolving-RL, a framework that optimizes how AI agents learn from past experiences to adapt to new tasks. The method jointly improves both experience extraction and utilization through reinforcement learning, achieving significant performance gains on out-of-distribution tasks without requiring test-time experience accumulation.

Evolving-RL addresses a fundamental limitation in current large language model deployments: their static nature prevents meaningful adaptation to novel situations. While LLMs excel at general tasks, they struggle with domain-specific adaptation when encountering unfamiliar problems. This research proposes a unified approach to self-evolution by treating experience extraction and utilization as complementary processes that should be optimized together rather than separately.

The significance lies in the coordinated co-evolution mechanism. Previous approaches focused either on how experiences are stored and represented or on how models use existing experiences. Evolving-RL bridges this gap by using evaluation signals to simultaneously improve both the extractor (which identifies reusable patterns) and the solver (which applies those patterns). The experimental results are substantial: up to 98.7% relative improvement on unseen ALFWorld tasks and 35.8% on Mind2Web demonstrates the framework's effectiveness at generalization.

For the AI development community, this work has important implications. The framework achieves performance gains even without accumulating experiences at test time, suggesting that reusable patterns can be internalized directly into model parameters through training. This makes deployed systems more efficient and reduces computational overhead. The approach also indicates a path toward more autonomous AI systems that genuinely adapt and improve rather than remaining static.

Looking ahead, the framework's effectiveness on diverse environments suggests broader applicability. Future work may expand this approach to multimodal tasks, larger-scale deployments, and real-world applications requiring genuine adaptation. The internalization of experience patterns could also inform how foundation models are fine-tuned for specialized domains.

• →Evolving-RL jointly optimizes experience extraction and utilization through coordinated co-evolution using dual supervisory signals
• →Framework achieves 98.7% relative improvement on ALFWorld unseen tasks by enabling genuine adaptation to novel situations
• →Reusable experience patterns internalized into model parameters reduce reliance on test-time experience accumulation
• →Method addresses the static nature of LLMs by distilling actionable insights from past interactions for deployment-time adaptation
• →Coordinated optimization of both components proves essential—performance gains only fully manifest through joint evolution