A controlled study comparing three AI scaffolding approaches across five large language models reveals that prompt engineering and system design choices can swing accuracy by up to 28 percentage points on the same task, challenging assumptions that published capability scores reflect true model performance and suggesting the elicitation gap persists even as models improve.
This research addresses a critical blind spot in AI capability evaluation: the conflation of what models can actually do versus what their scaffolding helps them accomplish. The study's controlled methodology—holding tasks and conditions fixed while varying only the prompt structure—isolates the pure effect of engineering choices from underlying model capability. The finding that a single model's accuracy can shift by 28 percentage points depending on whether it uses ReAct, multi-agent planning, or sequential planning undermines confidence in published benchmarks that report static capability scores without specifying scaffold details.
The counterintuitive finding that more capable models (Claude Opus) benefit most from structured scaffolds at harder difficulty levels inverts conventional expectations. This suggests capability and scaffold-sensitivity aren't inversely related; instead, sophisticated models may better leverage well-designed systems for complex reasoning. The fact that model family rather than absolute capability tier determined which scaffolds worked best indicates that architectural differences between providers create fundamentally different compatibility patterns with different reasoning structures.
For the AI evaluation ecosystem, this introduces methodological urgency. Current benchmarks risk systematically misrepresenting model capabilities by implicitly optimizing for particular scaffolds without transparency. The discovery that Gemini with planner-then-executor achieves both lowest cost and highest accuracy at Level 2 suggests practical optimization paths exist but require explicit experimental discovery rather than assumption. The finding that the elicitation gap doesn't necessarily shrink as models improve means future capability gains may remain largely inaccessible unless scaffolding techniques advance in parallel, creating a disconnect between raw model improvement and measured performance.
- →Scaffold choice alone can produce 28-percentage-point accuracy swings on identical tasks within single models, invalidating non-scaffold-specific capability claims
- →More capable models show highest scaffold sensitivity at harder difficulty levels, contradicting assumptions that improved models need less engineering assistance
- →Model family rather than capability tier determines scaffold compatibility, suggesting provider-specific reasoning architecture differences matter more than raw power
- →Current published capability scores conflate model ability with engineering optimization, creating an unmeasured and undisclosed elicitation gap
- →Better models don't automatically narrow the gap between their true capability and measured performance without corresponding improvements in reasoning scaffolds