It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers

A controlled study of 432 experiments across six LLM models challenges the assumption that higher-capability models require less structural guidance. The research reveals non-monotone harness sensitivity patterns, where frontier models like Gemini 2.5 Flash show performance degradation with increased harness complexity, while reasoning-focused models benefit from stricter constraints.

This research upends a foundational assumption in LLM agent deployment: that model capability and harness complexity maintain an inverse relationship. The 432-run experiment systematically tested six models across four capability tiers using light, balanced, and strict harness conditions on a 24-task synthetic benchmark with verifiable outputs. The findings reveal critical non-linearities that have substantial implications for how practitioners deploy large language models in production environments.

The harness-complexity paradox identified in Gemini 2.5 Flash demonstrates that increased structural guidance can actively degrade frontier chat model performance by 29-38 percentage points. This suggests that highly capable models may suffer from over-constrained instruction sets that interfere with natural problem-solving patterns. Conversely, the Qwen3.5-122B reasoning model achieved optimal performance (91.7% VTSR) under strict harness conditions while simultaneously reducing latency—a counterintuitive finding that indicates reasoning-oriented architectures benefit from explicit structural constraints.

The taxonomy of failure modes reveals mechanistic differences between capability tiers: capable models primarily fail on format violations while lower-capability models struggle with fundamental task execution errors like file placement. This distinction suggests that optimal harness design must account not just for model tier but fundamentally for model architecture and training objectives.

For practitioners, these findings translate to rejection of one-size-fits-all deployment strategies. The observation that a 2B model (Gemma4:e2B) matched frontier performance stability across all harness conditions suggests efficiency gains without systematic reliability trade-offs. Organizations deploying LLM agents must conduct model-specific harness optimization rather than assuming architectural scaling laws determine guidance requirements.

• →Harness sensitivity across LLM models is non-monotone, refuting the assumption that higher-capability models universally need less structural guidance.
• →Frontier chat models like Gemini 2.5 Flash experience 29-38 percentage point VTSR degradation with increased harness verbosity, suggesting over-constraint harms performance.
• →Reasoning-focused models like Qwen3.5-122B achieve optimal results with strict harnesses while reducing latency, opposite to monotone relationship predictions.
• →Failure mode analysis shows capable models fail primarily on format violations while lower-capability models struggle with fundamental execution errors like wrong file placement.
• →Practical deployment requires model-specific harness optimization rather than capability-tier-based assumptions about structural guidance requirements.