Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guarantees

Researchers introduced SLARouter, an online algorithm that optimizes LLM request routing by learning cost-efficient policies from sparse user feedback while guaranteeing Service Level Agreement compliance. The approach reduces operating costs by up to 2.2x compared to existing solutions without requiring per-benchmark tuning.

The escalating inference costs of large language models present a critical business challenge for companies deploying LLM applications at scale. SLARouter addresses this tension by enabling systems to route requests intelligently across models of varying cost and quality, ensuring contractual service guarantees while minimizing expenses. This matters because production LLM systems must balance competing pressures: users demand high-quality responses, commercial agreements legally mandate uptime and response standards, yet infrastructure costs grow with computational demand.

The innovation builds on emerging trends in cost-aware LLM optimization, but distinguishes itself through practical design choices. Unlike prior approaches requiring complete feedback data, offline training phases, or extensive per-workload configuration, SLARouter operates in real-time with the incomplete feedback naturally available in production environments—users typically signal dissatisfaction, not satisfaction. The algorithm provides formal theoretical guarantees for both cost minimization and strict SLA compliance, addressing a critical gap where many optimization techniques lack binding assurances.

For businesses deploying LLM applications, cost reduction of 2.2x directly impacts unit economics and profitability margins. Infrastructure teams gain a generalizable solution that adapts across different benchmarks without manual tuning, reducing operational complexity. The approach also signals a broader shift in AI economics toward practical, production-aware algorithms rather than theoretical frameworks assuming ideal conditions.

Looking forward, similar adaptive routing techniques may become standard infrastructure for multi-model deployments. Success of this approach could accelerate adoption of smaller, cheaper models for suitable tasks, potentially reshaping which models capture commercial demand and raising questions about how model quality preferences evolve when cost constraints are dynamically managed.

• →SLARouter reduces LLM operating costs by up to 2.2x while maintaining Service Level Agreement compliance without per-workload tuning
• →The algorithm learns optimal routing policies from sparse, one-sided user feedback typical of production systems rather than complete feedback data
• →Theoretical guarantees for both cost optimality and strict SLA compliance address critical gaps in existing cost-aware routing approaches
• →Online learning enables real-time adaptation to changing conditions, eliminating the offline training requirements of prior methods
• →Practical design for production environments positions this approach as potentially industry-standard infrastructure for multi-model LLM deployments