StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns

Amazon researchers introduced StaminaBench, a benchmark that evaluates coding agents' ability to handle extended multi-turn interactions (up to 100 consecutive change requests), revealing that current LLMs fail within 5-6 turns and that test feedback can improve performance up to 12x.

StaminaBench addresses a critical gap in AI evaluation methodologies by shifting focus from traditional task-completion metrics to real-world coding scenarios where agents must maintain consistency and correctness across dozens or hundreds of sequential modifications. This research matters because it exposes significant limitations in current large language models when deployed for extended development sessions—a common requirement in actual software engineering workflows where iterative refinement is standard practice.

The benchmark's design represents an important methodological advance. By generating tests programmatically without LLM involvement and running agents in isolated, language-agnostic environments communicating via HTTP, the researchers ensure reproducibility and eliminate confounding variables that plague many AI benchmarks. Testing six agent harnesses with seven open-source LLMs across 20 scenarios reveals that architectural choices and feedback mechanisms dramatically influence performance, with stronger models showing up to 6x performance variance depending on the harness implementation.

The findings have substantial implications for the AI development community. The discovery that test feedback loops improve performance by up to 12x suggests that robust error-handling and iterative correction mechanisms should be prioritized in production AI coding systems. The stark reality that all tested models fail within 5-6 turns indicates that current approaches to multi-turn reasoning are fundamentally inadequate for production coding tasks, requiring architectural innovations beyond scaling model parameters.

Looking forward, developers should monitor how research teams implement the feedback mechanisms identified in this work and watch for harness designs that can close the performance gap between models. The release of StaminaBench as an open-source resource will likely accelerate research into multi-turn agent reliability, which remains critical for practical AI-assisted development tools.

• →All tested LLMs fail within 5-6 interaction turns, revealing severe limitations in sustained reasoning for coding tasks.
• →Test feedback mechanisms improve performance by up to 12x, suggesting error detection and correction are essential for multi-turn agents.
• →Harness architecture matters more than model strength, with 6x performance gaps between best and worst configurations for the same model.
• →StaminaBench's black-box, language-agnostic testing approach provides a reproducible methodology for evaluating real-world coding agent behavior.
• →Current AI coding agents are inadequate for production environments requiring extended iterative development cycles.