Syntax- and Compilation-Preserving Evasion of LLM Vulnerability Detectors

Researchers demonstrate that LLM-based vulnerability detectors, increasingly used in software security pipelines, can be evaded through syntax-preserving code transformations. The study reveals that models with 70%+ accuracy on clean code can fail to detect 87%+ of vulnerabilities when subjected to minor edits, with adversarial attacks achieving up to 92.5% evasion rates—raising serious questions about the reliability of AI-driven security tools in production environments.

This research exposes a critical vulnerability in the deployment of large language models as security gatekeepers in CI/CD pipelines. While LLMs have shown promise in detecting code vulnerabilities, the study demonstrates they rely on superficial pattern matching rather than deep semantic understanding. Researchers tested five attack variants across four code transformation families on 5,000 C/C++ samples, finding that behavior-preserving edits—changes that don't alter program logic—consistently bypass detection. The concerning metric is Complete Resistance (CR), which measures vulnerabilities that survive all attack variants; models achieving strong benchmark performance showed CR as low as 0.12%, meaning attackers only need one successful evasion technique per vulnerability.

The transferability of these attacks amplifies the problem. Universal adversarial strings optimized on a 14-billion-parameter surrogate model transfer effectively to proprietary black-box APIs including GPT-4o, suggesting no vendor is immune. On-target optimization further pushes evasion rates to 92.5%, implying determined attackers can achieve near-perfect bypass rates. This finding directly challenges the assumption that benchmark accuracy correlates with real-world security robustness.

For the software development industry, this research indicates that relying on LLM-based security tools as a primary defense creates false confidence. Organizations deploying these detectors in critical security gates may unknowingly allow vulnerable code into production. The gap between clean-accuracy metrics (70%+) and actual robustness (potentially 0.12% CR) reveals a measurement crisis in AI security tooling. Future development must prioritize adversarial robustness testing and potentially hybrid approaches combining multiple detection methods rather than singular LLM-based gating.

• →LLM vulnerability detectors can be evaded through syntax-preserving code edits in over 87% of detected cases despite high benchmark accuracy
• →Universal adversarial strings transfer across different models, including GPT-4o, making black-box APIs equally vulnerable to white-box attacks
• →Complete Resistance metric reveals a massive gap between clean accuracy (70%+) and actual evasion resistance (as low as 0.12%)
• →Benchmark accuracy alone is insufficient to guarantee security for deployed vulnerability detectors in production environments
• →On-target adversarial optimization achieves up to 92.5% attack success rate, demonstrating severe robustness limitations