CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMs

Researchers introduce CalBench, a controlled evaluation framework for testing multi-agent LLM coordination in calendar scheduling scenarios where agents must negotiate shared commitments while protecting private information. The benchmark measures coordination quality, communication efficiency, fairness, and privacy leakage in decentralized systems where no single agent has complete information.

CalBench addresses a fundamental challenge in multi-agent AI systems: how to achieve effective coordination when information is distributed and sensitive. The benchmark simulates real-world constraints where agents manage private calendars and must schedule meetings collectively without exposing unnecessary private details. This represents an important departure from simpler multi-agent benchmarks where one capable agent can often solve problems independently, making CalBench uniquely suited for studying genuinely decentralized coordination.

The framework's significance stems from the increasing deployment of AI agents in privacy-sensitive domains like enterprise scheduling, healthcare coordination, and financial negotiations. As organizations move toward agent-based systems, the ability to verify both coordination effectiveness and privacy preservation becomes critical. CalBench's oracle-based approach enables precise measurement of coordination quality against optimal solutions, while its Distributed Constraint Optimization baseline ensures fair comparison under identical information constraints.

For AI researchers and developers, CalBench provides a standardized evaluation environment for testing negotiation protocols, communication strategies, and privacy-preserving mechanisms. This is particularly relevant as enterprises deploy multi-agent systems in regulated industries where privacy compliance is mandatory. The benchmark's emphasis on fairness in cost distribution also addresses growing concerns about equitable outcomes in automated coordination systems.

Looking ahead, CalBench could influence how multi-agent AI systems are evaluated and deployed in production environments. As LLM-based agents become more prevalent in enterprise workflows, benchmarks that verify both capability and privacy compliance will become essential for adoption. The framework may inspire similar evaluation environments for other coordination-intensive domains.

• →CalBench enables precise measurement of multi-agent coordination quality through calendar scheduling with oracle-generated optimal solutions
• →The benchmark uniquely requires decentralized decision-making where no agent accesses others' private calendars, better reflecting real-world constraints
• →Framework measures coordination efficiency, communication overhead, fairness in cost distribution, and unintended privacy information leakage
• →DCOP baseline ensures fair comparison of LLM coordination against deterministic algorithms under identical information constraints
• →Results inform development of privacy-preserving negotiation strategies for AI agents in regulated enterprise environments