PrivacyPeek: Auditing What LLM-Based Agents Acquire, Not Just What They Say

PrivacyPeek introduces a new benchmark for evaluating privacy vulnerabilities in LLM-based agents, revealing that autonomous AI systems routinely acquire sensitive information beyond what tasks require. The research demonstrates that existing privacy audits miss critical acquisition-stage leakage, where data enters the agent's context, and that current prompt-level defenses are largely ineffective.

The emergence of autonomous LLM-based agents has created a significant blind spot in privacy assessment methodologies. While organizations focus on auditing what agents disclose in their outputs and actions, PrivacyPeek identifies a more fundamental vulnerability: the acquisition stage where sensitive data first enters an agent's context window. This distinction matters because once information is acquired, it exists in memory as a potential attack surface, vulnerable to follow-up prompts or system compromises even if never initially disclosed.

This research reflects a broader pattern in AI development where capability gains outpace security implementations. As agents become more autonomous and task-capable, they increasingly request access to broader datasets to improve performance, creating what researchers call the capability-leakage correlation. The benchmark's evaluation of 10 LLM-based agents across 1,182 test cases demonstrates the problem is systemic rather than isolated to specific models.

The findings have direct implications for enterprise deployment of AI agents in sensitive domains like healthcare, finance, and legal services. Organizations implementing these systems face reputational and regulatory risk, as data acquisition patterns may violate privacy frameworks like GDPR even if the sensitive information is never disclosed. The ineffectiveness of prompt-level defenses suggests the problem requires architectural solutions rather than instruction-based mitigations.

The release of the PrivacyPeek dataset and code establishes new evaluation standards for the AI development community. Future work likely involves developing acquisition-minimization techniques, sandboxing strategies, and architectural changes that decouple agent capabilities from data access breadth. This research accelerates the timeline for privacy-aware agent design becoming a competitive necessity.

• →LLM-based agents acquire sensitive information beyond task requirements, creating security vulnerabilities even when output is filtered
• →Existing privacy benchmarks miss acquisition-stage leakage, the earliest point where sensitive data enters agent context
• →Prompt-level defenses mitigate only a small fraction of acquisition-stage privacy risks, indicating architectural solutions are necessary
• →A positive correlation exists between task-completion capability and privacy leakage, forcing developers to choose between performance and privacy
• →The benchmark dataset establishes new evaluation standards for assessing agent privacy across 16 application domains