SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior

Researchers introduce SkillJuror, a framework measuring how LLM agent skill organization affects runtime behavior independent of content. Testing Progressive Disclosure—a hierarchical skill structure—against flat baselines shows agents access 3.26x more resources and achieve 4.1% higher verification rates, revealing that procedural knowledge presentation meaningfully influences agent reasoning patterns.

SkillJuror addresses a fundamental gap in AI agent evaluation: distinguishing skill *content* from skill *organization*. Prior benchmarks treat these as inseparable, making it impossible to isolate whether performance improvements stem from what information is provided or how it is structured. This research isolates organization's effect using semantic controls and matched multi-trial evaluation, a methodologically rigorous approach that strengthens confidence in findings.

The core insight—Progressive Disclosure increases resource discovery from 1.18 to 3.85 touches per trajectory—suggests LLM agents navigate hierarchical structures more effectively than flat repositories. This mirrors cognitive science principles about information retrieval and decision-making. The 4.1% aggregate improvement, while modest, represents consistent gains across 410 matched trials, indicating reliable organizational effects rather than statistical noise.

For AI developers building production systems, this work validates the intuition that skill architecture matters. However, the task-dependent nature of benefits matters critically: gains appear strongest when resources guide implementation or error recovery, but vanish when tasks demand precise output formatting or numerical exactness. This boundary condition prevents over-generalizing the findings and suggests Progressive Disclosure works by improving *exploration* of solution pathways rather than inherently fixing output precision issues.

The framework itself—semantic variant generation and trajectory analysis—establishes methodology that future research can build upon. As agent systems scale to handle increasingly complex multi-step tasks, understanding how knowledge organization shapes exploration behavior becomes economically significant for model efficiency and reliability.

• →Skill organization meaningfully changes LLM agent runtime behavior independent of content, challenging assumptions in current benchmarks.
• →Progressive Disclosure increases resource discovery 3.26x and yields 4.1% higher verification rates over flat baselines in 82-task evaluation.
• →Organization benefits are task-dependent: strongest for implementation guidance and error recovery, weaker for strict output conventions.
• →SkillJuror's methodology using semantic variants and trajectory evidence provides reusable framework for isolating skill architecture effects.
• →Findings suggest efficiency gains from better skill design could reduce token consumption and improve agent reasoning transparency in production systems.