Decoupling Reconnaissance and Exploitation: Measuring the Capability Boundaries of LLM-Based Web Penetration Testing

Researchers propose a decoupled evaluation framework for testing LLM-based penetration testing agents by separating reconnaissance from exploitation tasks. The study reveals significant capability gaps: agents achieve 90% success with accurate vulnerability context but only 50% autonomous reconnaissance performance, with distinct strengths across different architectural designs.

This research addresses a critical blind spot in evaluating LLM-based security automation tools. Traditional end-to-end testing conflates two distinct challenges—finding vulnerabilities and exploiting them—making it impossible to identify where agents actually fail. The decoupled framework isolates these stages using ground-truth injection, providing clearer insight into LLM capabilities for offensive security work.

The findings expose a substantial performance cliff between guided and autonomous operation. When researchers provide accurate vulnerability context, exploitation success rates reach 90%, demonstrating that LLMs can execute complex attack sequences effectively. However, autonomous reconnaissance—where agents must independently identify and parse security data from unstructured telemetry—plateaus at roughly 50%, suggesting this remains the genuine bottleneck. This gap reflects a broader challenge in AI systems: executing known procedures versus discovering unknown problems.

The architectural analysis reveals nuanced trade-offs. Multi-agent systems excel at complex, sequential exploits like deserialization attacks where task decomposition helps manage state. Monolithic designs perform better on direct injection attacks, while graph-driven architectures handle cross-session vulnerabilities more effectively. These findings matter for security tool developers deciding on system design patterns for automated penetration testing platforms.

For the broader AI security landscape, this research establishes that current LLM-based agents require human reconnaissance input to be practically effective. Organizations deploying such tools should expect them to augment rather than replace human testers during vulnerability discovery phases. Future development should prioritize improving unstructured data parsing and contextual inference to close the reconnaissance gap.

• →LLM-based penetration testing agents achieve 90% exploitation success with accurate vulnerability context but only 50% autonomous reconnaissance performance
• →Failures in parsing unstructured telemetry represent the primary bottleneck preventing autonomous vulnerability discovery
• →Multi-agent architectures excel at complex sequential exploits while monolithic and graph-driven designs perform better on simpler injection attacks
• →Decoupled evaluation frameworks provide more accurate capability assessment than end-to-end testing by isolating reconnaissance from exploitation stages
• →Current LLM agents require human-provided vulnerability context to be practically effective for real-world penetration testing