Probabilistic Agents in Deterministic Audits: Evaluating Multi-Agent Systems for Automated Audits Based on the German IT-Grundschutz

Researchers present a Multi-Agent System architecture using Hybrid Retrieval Augmented Generation to automate IT-Grundschutz compliance auditing, addressing the resource-intensive certification burden mandated by the NIS-2 Directive. While the system excels at semantic tasks like structural analysis and modeling, it struggles with deterministic logical reasoning phases due to the probabilistic nature of current large language models.

The NIS-2 Directive creates unprecedented compliance pressure on thousands of small and medium enterprises across Europe, requiring robust security audits against established frameworks like Germany's IT-Grundschutz standard. Traditional certification remains prohibitively expensive and labor-intensive, creating a critical gap between regulatory requirements and organizational capacity to implement them at scale.

This research addresses that gap by deploying multi-agent AI systems to partially automate the audit process. The researchers introduced two key technical innovations: a Hypothesis-Verification Loop that cross-references AI-generated dependencies against knowledge graphs to reduce hallucinations, and a Decoupled Reasoning Pipeline separating semantic extraction from logical inheritance rules. Using the BSI's RecPlast GmbH case study as a reference benchmark, the team evaluated performance across all major audit phases.

The empirical findings reveal a fundamental challenge in applying probabilistic AI systems to deterministic compliance frameworks. The architecture performs well in semantic-heavy phases—structural analysis and modeling—where pattern recognition and information extraction naturally suit LLM strengths. However, protection needs assessment and compliance checks require absolute logical accuracy that probabilistic systems cannot guarantee, creating a hard ceiling on automation potential.

For the compliance and cybersecurity industry, this work highlights both opportunity and limitation. Organizations can leverage AI to reduce administrative overhead in documentation and initial analysis phases, potentially lowering adoption barriers for NIS-2 compliance. However, the deterministic nature of security risk assessment means human expert review remains non-negotiable for certification validity, limiting full automation benefits and maintaining demand for specialized auditors.

• →Multi-agent systems show promise for automating semantic phases of IT-Grundschutz audits but cannot replace human oversight in deterministic logical reasoning tasks.
• →NIS-2 Directive compliance costs create strong market incentive for partial automation solutions despite current LLM limitations.
• →Hybrid RAG architectures with knowledge graph verification significantly reduce AI hallucinations in compliance contexts.
• →Probabilistic AI models fundamentally struggle with binary compliance requirements that demand deterministic certainty.
• →Audit automation potential is phased—high-impact in documentation and analysis, limited in final validation and certification decisions.