Ekka: Automated Diagnosis of Silent Errors in LLM Inference

Researchers introduce Ekka, an automated diagnostic system that identifies root causes of silent errors in large language model serving frameworks by comparing execution states between target and reference implementations. The system achieves 80% pass@1 accuracy and has already discovered 4 new bugs in production serving frameworks, addressing a critical reliability challenge in LLM deployment.

Silent errors in LLM inference represent a fundamental reliability challenge in production AI systems. When output quality degrades without explicit error signals, developers face an extremely difficult debugging landscape where symptoms manifest at the semantic level while root causes hide in complex software stacks. Ekka transforms this diagnostic problem by treating it as differential debugging—leveraging semantically correct reference implementations as ground truth to systematically identify where divergence occurs. This approach is particularly valuable given the rapid evolution of LLM serving frameworks like vLLM, TensorRT-LLM, and others, where aggressive optimizations frequently introduce subtle bugs. The benchmark results demonstrate substantial practical utility: 80% accuracy on first attempt and 88% within five attempts represents a significant improvement over existing debugging methodologies. The discovery of four confirmed bugs in production frameworks validates the system's real-world applicability. For the AI infrastructure ecosystem, this work addresses a critical pain point affecting inference reliability and cost efficiency. Organizations deploying large-scale LLM services face substantial losses from degraded output quality that goes undetected. Ekka's automated approach could substantially reduce the time and expertise required to maintain serving framework integrity. Looking forward, the adoption of such diagnostic tools may accelerate confidence in optimized serving frameworks, enabling faster deployment of performance improvements. The research also suggests broader opportunities for automated testing frameworks tailored to LLM-specific failure modes, which remain underexplored compared to traditional software testing methodologies.

• →Ekka automates diagnosis of silent errors in LLM serving frameworks using differential debugging between reference and target implementations.
• →System achieves 80% first-attempt accuracy in identifying root causes, outperforming existing diagnostic approaches.
• →Four previously unknown bugs discovered and confirmed in production serving frameworks demonstrate real-world validation.
• →Addresses critical reliability gap in LLM inference where output degradation occurs without explicit error signals.
• →Differential debugging methodology establishes new pattern for testing complex AI infrastructure software stacks.