OrcaRouter: A Production-Oriented LLM Router with Hybrid Offline-Online Learning

OrcaRouter is a production-ready LLM routing system that uses contextual bandits and hybrid offline-online learning to intelligently direct requests to the most appropriate language model. The system ranked second on the RouterArena leaderboard with 75.54% accuracy while maintaining low inference costs of $1.00 per 1,000 queries.

OrcaRouter addresses a critical infrastructure challenge in the multi-model LLM landscape: efficient request routing. As organizations deploy diverse language models with varying capabilities, latencies, and costs, intelligent routing becomes essential for optimizing both performance and expenses. The system's LinUCB-based contextual bandit approach represents a sophisticated solution that learns from both curated offline evaluations and real-world deployment feedback, creating a feedback loop that improves over time.

The hybrid offline-online learning protocol is particularly noteworthy. The offline phase establishes baseline performance metrics by evaluating candidate models against curated routing prompts, generating a reward matrix that initializes ridge regressors for each model. This pre-training prevents cold-start problems and ensures reasonable performance from day one. During deployment, OrcaRouter can continue learning from actual user feedback, gradually optimizing routing decisions based on real-world patterns rather than purely synthetic evaluations.

The RouterArena leaderboard ranking demonstrates OrcaRouter's competitive viability in production environments. Achieving 75.54% accuracy while maintaining sub-cent costs per query suggests the system balances accuracy with economic efficiency—a critical requirement for scalable AI infrastructure. This positions OrcaRouter as relevant to organizations managing heterogeneous model portfolios seeking to maximize resource utilization.

Future development will likely focus on expanding feature engineering beyond lexical and sentence-embedding representations, potentially incorporating semantic complexity analysis or task-specific signals. As model diversity increases and inference costs remain a primary concern, sophisticated routing infrastructure becomes increasingly valuable to the broader AI deployment ecosystem.

• →OrcaRouter uses contextual bandits with offline-online learning to route requests to optimal LLMs based on capabilities and costs
• →The system ranked second on RouterArena with 75.54% accuracy while achieving $1 per 1,000 query costs
• →Hybrid learning approach combines curated offline evaluations with real-world deployment feedback for continuous improvement
• →Ridge regression models per arm enable efficient parameter updates without retraining entire routing systems
• →Production-grade routing infrastructure is becoming essential for cost-effective multi-model LLM deployments