Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?

A research study challenges the widespread practice of using context files (like AGENTS.md) to enhance coding agent performance, finding that these files provide no measurable improvement in task completion rates while increasing inference costs by over 20%. The findings suggest that while context files help agents follow instructions, repository overviews—commonly recommended by model providers—offer minimal practical value.

The study addresses a fundamental assumption in AI-assisted software development: that providing structured repository context improves coding agent performance. Researchers evaluated this hypothesis across multiple dimensions, testing both LLM-generated and developer-authored context files on established benchmarks and real-world repository issues. The counterintuitive results reveal a significant gap between industry best practices and empirical reality.

This research emerged from growing adoption of coding agents across enterprises, where companies invest substantial effort in crafting detailed AGENTS.md files and similar documentation. Model providers actively encourage this practice, suggesting it enhances agent capabilities. The study's rigorous evaluation methodology—spanning different LLMs and agent architectures—provides credible evidence that this guidance may be misaligned with actual performance outcomes.

The implications extend beyond academic interest. Development teams currently spending resources to create and maintain context files face questions about return on investment. The 20% average increase in inference costs translates to meaningful operational expenses for organizations running coding agents at scale. Furthermore, the finding that instructions are followed effectively while repository overviews fail to improve outcomes suggests a critical distinction: agents execute directives well but struggle to leverage abstract repository information productively.

Looking forward, this research should prompt model developers and enterprise adopters to reassess their documentation strategies. Organizations might redirect efforts from comprehensive repository overviews toward specific instruction sets. The findings also highlight the need for similar empirical evaluations of other widely-assumed best practices in AI development workflows, potentially uncovering additional efficiency improvements or cost-saving opportunities.

• →Context files like AGENTS.md increase inference costs by over 20% without improving task completion success rates
• →Repository overviews recommended by model providers prove ineffective despite widespread endorsement
• →Coding agents successfully follow explicit instructions in context files but fail to benefit from abstract documentation
• →The gap between industry best practices and empirical performance suggests many AI development assumptions need rigorous validation
• →Organizations should prioritize specific, actionable directives over comprehensive repository documentation in agent prompts