Capturing LLM Capabilities via Evidence-Calibrated Query Clustering

Researchers introduce ECC (Evidence-Calibrated Query Clustering), an algorithm that improves how AI systems evaluate large language model capabilities by organizing queries into groups that reflect actual performance requirements rather than surface-level semantics. The method outperforms existing clustering approaches by 17-18 percentage points and shows practical value in downstream applications like query routing.

ECC addresses a fundamental challenge in LLM evaluation: the gap between what queries appear to ask and what capabilities they actually demand. Traditional clustering methods rely on semantic embeddings or taxonomies that often misalign with true capability requirements, leading to inaccurate capability assessments. This research proposes a calibration approach that refines semantic embeddings using limited model comparisons, creating capability-aware clusters parameterized by Bradley-Terry models—a statistical framework for ranking from pairwise comparisons. The algorithm accommodates queries with mixed capability demands through trainable mixture weights, enabling more nuanced capability profiling.

The advancement matters because accurate LLM capability assessment underpins better system design, benchmarking, and deployment decisions. As organizations increasingly rely on LLMs for critical tasks, understanding their true capabilities—rather than assumed ones—becomes essential for risk management and performance optimization. Current embedding-based approaches fail because they prioritize linguistic similarity over functional requirements; ECC bridges this gap by incorporating actual model behavior into the clustering process.

For developers and AI teams, this approach enables more intelligent query routing and capability-aware inference, reducing computational costs by directing queries to appropriately-sized models. The 17-18 percentage point improvements over baselines suggest meaningful gains in ranking quality that could translate to better system decisions in production environments. The methodology's effectiveness in downstream applications indicates practical value beyond theoretical contribution, positioning capability-aware clustering as increasingly important for managing complex LLM systems.

• →ECC calibrates semantic embeddings using model comparisons to align query clustering with actual LLM capability demands rather than surface-level semantics.
• →The algorithm outperforms human-labeled and embedding-based baselines by 17.64 and 18.02 percentage points respectively in capability ranking quality.
• →Bradley-Terry parameterization and trainable mixture weights enable nuanced capability profiling for queries with mixed demands.
• →Practical applications include improved query routing and capability-aware inference in production LLM systems.
• →The approach bridges the critical gap between linguistic similarity and functional capability requirements in LLM evaluation.