Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models

Researchers conducted a reproducibility study of Vul-RAG, a RAG-based framework for detecting software vulnerabilities using LLMs, and found that while results are reproducible with open-weight models, performance plateaus around 0.30 pairwise accuracy regardless of model sophistication. The findings suggest that simply scaling up model capacity does not substantially improve vulnerability detection capabilities.

This reproducibility study addresses a critical gap in AI security research by testing whether vulnerability detection systems actually work beyond controlled proprietary environments. The Vul-RAG framework represents an important application of retrieval-augmented generation to practical security problems, but the authors' findings reveal fundamental limitations that the field must confront. By evaluating diverse open-weight models ranging from specialized code models to general-purpose systems, the research demonstrates that the reported improvements in Vul-RAG do transfer to local deployments, validating the original methodology while identifying its inherent constraints.

The persistent performance plateau at 0.30 pairwise accuracy across all tested models carries significant implications for AI-powered security tooling. This ceiling suggests that architectural limitations or training data constraints may be more restrictive than previously assumed, indicating that throwing larger or more capable models at the problem yields diminishing returns. The finding contrasts sharply with broader AI trends where scaling has typically driven performance improvements, suggesting vulnerability detection may require fundamentally different approaches rather than incremental model improvements.

For developers and security teams, these results inject realism into expectations around AI-assisted code security. Organizations cannot rely on future model releases to automatically solve vulnerability detection at scale. Instead, the research points toward hybrid approaches that combine model limitations with complementary techniques. The public availability of implementation artifacts enables the security community to build upon this foundation and explore alternative frameworks. Looking forward, the plateau effect should prompt researchers to investigate whether the problem lies in model training, retrieval augmentation strategies, or the fundamental difficulty of vulnerability semantics itself.

• →Vul-RAG vulnerability detection results are reproducible with open-weight models but show consistent performance plateaus around 0.30 pairwise accuracy.
• →Larger and more advanced LLMs do not substantially improve vulnerability detection performance, suggesting architectural rather than capacity-based limitations.
• →Reproducibility research reveals that model choice alone cannot overcome fundamental constraints in current RAG-based security approaches.
• →The study includes comprehensive evaluation across code-specialized, general-purpose, and reasoning models of varying sizes to establish generalizability.
• →Public implementation artifacts enable further research into alternative vulnerability detection methodologies beyond simple model scaling.