SAGE: A Service Agent Graph-guided Evaluation Benchmark

Researchers introduce SAGE, a comprehensive benchmark for evaluating Large Language Models in customer service automation that uses dynamic dialogue graphs and adversarial testing to assess both intent classification and action execution. Testing across 27 LLMs reveals a critical 'Execution Gap' where models correctly identify user intents but fail to perform appropriate follow-up actions, plus an 'Empathy Resilience' phenomenon where models maintain polite facades despite underlying logical failures.

SAGE addresses a fundamental gap in LLM evaluation methodologies by moving beyond static, single-metric benchmarks toward dynamic, real-world assessment frameworks. The research exposes critical weaknesses in how current LLMs handle customer service workflows, particularly the disconnect between intent understanding and action execution. This matters because companies deploying these models for automation must ensure both logical correctness and operational reliability, not merely conversational fluency.

The benchmark's formalization of Standard Operating Procedures into Dynamic Dialogue Graphs represents a significant methodological advance. By converting unstructured business rules into verifiable logical paths, SAGE enables comprehensive testing across diverse scenarios while maintaining reproducibility. The Adversarial Intent Taxonomy and modular extension mechanism allow cost-effective deployment across industries, addressing a practical bottleneck in enterprise AI evaluation.

For developers and enterprises, these findings carry substantial implications. The 'Execution Gap' suggests current LLMs require additional architectural safeguards or training approaches to ensure operational correctness beyond conversational quality. The 'Empathy Resilience' phenomenon—where models mask logical failures through politeness—highlights potential risks in customer-facing deployments where users might trust systems that are actually unreliable. This could expose organizations to service failures and customer dissatisfaction.

Looking forward, SAGE's open-source framework may drive industry standardization in LLM evaluation for service automation. The benchmark's findings will likely influence how enterprises architect AI-powered customer service systems, potentially spurring demand for specialized fine-tuning approaches and formal verification methods. Developers should monitor whether major LLM providers improve execution-layer performance based on these insights.

• →SAGE benchmark reveals an 'Execution Gap' where LLMs correctly classify intents but fail to execute appropriate subsequent actions in customer service scenarios
• →The benchmark formalizes unstructured SOPs into Dynamic Dialogue Graphs, enabling precise logical compliance verification across 27 tested LLMs
• →Models demonstrate 'Empathy Resilience', maintaining polite responses despite underlying logical failures when facing adversarial inputs
• →Testing across 6 industrial scenarios shows significant performance variations, suggesting current LLMs are not production-ready for autonomous customer service without additional safeguards
• →Open-source framework enables low-cost deployment across domains and may drive standardization in enterprise AI evaluation methodologies