$\tau$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge

Researchers introduced τ-Knowledge, a new benchmark for evaluating AI conversational agents in knowledge-intensive environments, specifically testing their ability to retrieve and apply unstructured domain knowledge. Even frontier AI models achieved only 25.5% success rates when navigating complex fintech customer support scenarios with 700 interconnected knowledge documents.

• →τ-Knowledge benchmark reveals significant limitations in current AI agents' ability to handle unstructured knowledge retrieval and application.
• →Frontier AI models achieved only ~25.5% pass rates in realistic fintech customer support workflows.
• →Agents struggle with retrieving correct documents from densely interlinked knowledge bases and reasoning over complex policies.
• →The benchmark addresses a gap in realistic evaluation of AI agents in long-horizon interactions with unstructured data.
• →Performance reliability degrades sharply over repeated trials, highlighting consistency issues in AI agent deployment.