From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

Researchers present a new evaluation protocol for AI pentesting agents that moves beyond simplified benchmarks to assess real-world vulnerability discovery capabilities. The framework combines structured ground-truth validation with LLM-based semantic matching and includes efficiency metrics, addressing a critical gap in how offensive security AI systems are currently measured.

Current benchmarks for AI pentesting agents focus on narrow, predefined objectives like capture-the-flag competitions or specific exploit reproduction, failing to capture the complexity of real-world security assessments. This research addresses a fundamental measurement problem: existing evaluation methods optimize for task completion in simplified environments rather than practical vulnerability discovery across diverse attack surfaces. The proposed protocol represents a meaningful evolution in AI security evaluation methodology.

The development reflects broader industry maturation in AI security tooling. As pentesting agents become more capable and organizations consider deployment in production environments, evaluation methods must evolve beyond academic metrics. The shift toward validated vulnerability discovery with bipartite resolution scoring acknowledges realistic ambiguity in security findings—a critical distinction from controlled laboratory settings.

For the security and AI communities, this work has operational significance. Organizations evaluating AI pentesting solutions will have more informative comparison frameworks, potentially accelerating adoption of legitimate security tools while raising standards for what constitutes reliable agent performance. The release of annotated ground truth and reproducible code enables standardization across the field.

The evaluation protocol's emphasis on stochastic agent evaluation and efficiency metrics suggests recognition that real-world pentesting involves trade-offs between thoroughness and resource consumption. Future developments likely include expanded test suites, integration with vulnerability management platforms, and standardized reporting formats that align pentesting agent outputs with industry practices.

• →New evaluation protocol shifts focus from task completion metrics to practical vulnerability discovery in complex, realistic environments.
• →Framework addresses critical gap where existing benchmarks fail to capture decision-making complexity required in authentic pentesting scenarios.
• →LLM-based semantic matching combined with structured ground-truth enables more accurate vulnerability identification under realistic conditions.
• →Released open-source tools and annotated ground truth support reproducibility and potential industry standardization of agent evaluation.
• →Protocol design incorporating efficiency metrics and stochastic evaluation reflects maturation toward operationally-relevant AI security assessment.