MuPPET: A Benchmark for Contextual Privacy of LLM Assistants in Multi-Party Conversations

Researchers introduced MuPPET, a benchmark testing privacy vulnerabilities in large language model assistants operating in multi-party conversations. The study reveals that LLMs leak significantly more sensitive information in group settings than in one-to-one interactions, with both frontier and smaller open-weight models showing substantial exposure risks that existing privacy defenses cannot adequately address.

The emergence of MuPPET addresses a critical blind spot in AI safety research: while single-user privacy protections for LLMs have received attention, multi-party contexts present structurally different challenges that existing benchmarks fail to capture. In group chats and collaborative environments, an LLM agent must treat all shared information as potentially sensitive to any group member, creating exponentially more constraints than bilateral conversations. The research demonstrates that models currently deployed in production environments—both cutting-edge frontier models and smaller open-weight variants favored for local deployment—exhibit substantially elevated privacy leakage in these scenarios.

This finding matters because sensitive data handling in multi-party contexts is increasingly common in enterprise and consumer applications. Group chats, collaborative workspaces, and team environments represent realistic deployment scenarios where privacy breaches could expose personal information to unintended recipients simultaneously. The fact that smaller open-weight models show worse performance is particularly concerning, as organizations often choose local deployment specifically to maintain privacy control over sensitive data.

The research indicates that current contextual privacy defenses provide incomplete mitigation—they reduce but don't eliminate leakage, introduce utility degradation, and fail to solve the underlying party-tracking problem where models struggle to maintain awareness of recipient-specific privacy constraints. For developers and enterprises, this creates immediate pressure to reevaluate deployment assumptions and privacy architectures. The benchmark itself provides a foundation for improved safety research, but the gap between current defenses and demonstrated vulnerabilities suggests substantial engineering work remains before these systems can be confidently deployed in genuinely sensitive multi-party scenarios.

• →LLMs leak substantially more private information in multi-party conversations than in one-to-one settings, revealing a previously unmeasured vulnerability class.
• →Both frontier models and smaller open-weight models demonstrate significant privacy exposure, with smaller models performing worse than larger counterparts.
• →Existing contextual privacy defenses only partially mitigate the problem and fail to address the core technical challenge of party-aware information control.
• →Multi-party privacy in LLM systems is structurally harder to control because every piece of shared information must be appropriate for every group recipient.
• →Organizations deploying LLMs in group chat and collaborative environments should reassess their privacy assumptions and may need architectural redesigns before production use.