Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

Researchers at arXiv present findings that challenge assumptions about LLM agent capabilities, revealing that a model's base performance doesn't predict its ability to self-evolve through harness updates. The study identifies two distinct capabilities—harness-updating and harness-benefit—with counterintuitive results suggesting mid-tier models benefit most from self-evolution while strong models benefit less.

This research addresses a fundamental question in LLM agent development: whether investing resources in more capable base models necessarily improves their ability to self-evolve and benefit from updates. The study examines harness self-evolution—the process where agents adapt external components like prompts, skills, and tools based on execution evidence—without modifying underlying model parameters.

The findings reveal a critical disconnection between raw model capability and self-evolution effectiveness. Harness-updating capability shows remarkable flatness across model tiers, meaning even smaller models like Qwen3.5-9B generate updates yielding comparable improvements to Claude Opus. This suggests the evolution mechanism itself may be relatively model-agnostic. More surprisingly, harness-benefit demonstrates non-monotonic behavior: weak-tier models gain minimal improvements, mid-tier models achieve maximum gains, and strong-tier models show diminishing returns.

The researchers identify specific failure modes in weak-tier models—namely inability to properly invoke relevant harness artifacts and inconsistent adherence to instructions—providing actionable insights for future training approaches. These findings reshape conventional assumptions about agent development strategy, suggesting resources allocated toward improving base model strength may yield lower returns than anticipated for self-evolving systems.

For practitioners building LLM agents, this work implies a more nuanced optimization approach. Rather than pursuing maximum base capability, developers should focus on mid-tier models with improved harness invocation mechanisms and instruction-following reliability. The research directs attention toward instruction-following fidelity and artifact activation as critical training targets, potentially offering better value than raw capability scaling.

• →Harness-updating capability remains surprisingly flat across model tiers, with smaller models producing updates comparable to much larger models
• →Harness-benefit follows non-monotonic patterns, with mid-tier models gaining most and strong-tier models showing diminishing returns
• →Weak-tier models fail through two mechanisms: inability to activate harness artifacts or inability to follow instructions faithfully
• →Investing capability budget in task-solving agents rather than evolution mechanisms may yield better practical returns
• →Improving harness invocation and long-horizon instruction-following in training could unlock greater self-evolution benefits