LeakDojo: Decoding the Leakage Threats of RAG Systems

LeakDojo is a new research framework that systematically evaluates security vulnerabilities in Retrieval-Augmented Generation (RAG) systems, revealing that stronger LLM instruction-following capabilities correlate with higher data leakage risks. The study benchmarks six attack methods across multiple LLMs and datasets, providing critical insights into how RAG databases can be exploited and suggesting that improvements in RAG faithfulness may paradoxically increase security vulnerabilities.

LeakDojo addresses a critical security gap in the rapidly expanding RAG ecosystem. As LLMs integrate external knowledge bases through RAG systems, these databases become attractive targets for adversarial attacks designed to extract proprietary or sensitive information. This research demonstrates that RAG vulnerabilities are not merely theoretical—they scale with model capability, creating a fundamental tension between functionality and security.

The framework's findings reveal three interconnected risk factors. Query generation attacks and adversarial instructions operate independently but compound when combined, following a multiplicative rather than additive pattern. This mathematical relationship provides defenders with a clearer model for understanding attack surface area. The observation that instruction-following capability drives leakage risk suggests that model optimization for user alignment paradoxically increases exploitation vulnerability—a counterintuitive insight with major implications for deployment strategies.

For developers and organizations deploying RAG systems, these findings create immediate practical considerations. Current RAG implementations often prioritize retrieval accuracy and response quality without adequately addressing information leakage. The research indicates that standard hardening approaches may be insufficient against sophisticated adversaries. The revelation that RAG faithfulness improvements introduce leakage risks suggests that security-hardened RAG architectures may require fundamental redesign rather than incremental patches.

The open-source LeakDojo framework enables security researchers and practitioners to evaluate their own RAG implementations against established attack vectors. Organizations must now factor leakage risk into their RAG deployment decisions, potentially limiting external knowledge base sensitivity or implementing additional isolation layers. As RAG adoption accelerates across enterprises, systematic vulnerability assessment becomes essential rather than optional.

• →LeakDojo benchmarks six RAG attacks across fourteen LLMs, revealing that query generation and adversarial instructions combine multiplicatively to enable information leakage.
• →Stronger instruction-following capabilities in LLMs directly correlate with increased RAG system vulnerability to extraction attacks.
• →Improvements in RAG faithfulness—making systems more accurate—can paradoxically increase security risks by providing better attack vectors.
• →The framework provides the first systematic evaluation methodology for RAG leakage, enabling organizations to assess their own system vulnerabilities.
• →Security considerations may become a limiting factor in RAG deployment decisions for organizations handling sensitive external knowledge bases.