Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

Researchers introduce Harness-Bench, a diagnostic benchmark that measures how software infrastructure—not just base models—affects LLM agent performance across realistic workflows. The study of 5,194 execution trajectories reveals substantial variation in agent capability depending on harness configuration, suggesting performance metrics should reflect model-harness pairings rather than models alone.

Harness-Bench addresses a critical gap in how AI agent capabilities are evaluated and reported. While benchmark suites have historically focused on base model performance, this research demonstrates that the execution layer—encompassing context management, tool integration, state tracking, and error recovery—significantly influences real-world outcomes. The study's 106 sandboxed tasks, derived from practical agent-use patterns and manually validated, provide a reproducible foundation that captures execution behaviors often abstracted away in traditional benchmarks.

This work reflects a broader industry maturation where LLM agents transition from research artifacts to production systems. As organizations deploy agents for concrete tasks requiring tool use and workspace modification, the infrastructure layer becomes as critical as model quality. Previous benchmarking approaches either compared complete systems (conflating model and harness effects) or held harness fixed, preventing systematic evaluation of configuration-level impacts.

The implications for developers and organizations are substantial. The finding that agent capability varies significantly across model-harness pairings suggests that selecting an agent solution requires evaluating the entire stack rather than pursuing the largest or most capable base model in isolation. This could reshape procurement decisions and encourage investment in robust harness infrastructure. The identification of execution-alignment failures—where model reasoning decouples from tool feedback and workspace state—points to specific architectural vulnerabilities requiring attention.

Moving forward, standardized harness evaluation metrics could become industry practice, similar to how transformer architecture standardization preceded modern LLM development. Organizations should expect future AI infrastructure vendors to prioritize harness transparency and auditability alongside model performance.

• →Agent performance depends critically on harness configuration, not base model alone, requiring evaluation at the model-harness pairing level
• →5,194 execution trajectories reveal substantial variation in completion rates, efficiency, and failure patterns across different harness-model combinations
• →Execution-alignment failures occur when model reasoning becomes decoupled from tool feedback, workspace state, and verifiable outputs
• →Harness-Bench's 106 realistic, manually-validated tasks provide reproducible evaluation methodology for agent execution stacks
• →Results suggest future agent procurement and benchmarking should prioritize infrastructure transparency alongside base model capabilities