Litmus: Zero-Label, Code-Driven Metric Specification for Evaluating AI Systems

Researchers introduce Litmus, a zero-label evaluation system that automatically designs metrics for AI pipelines by analyzing source code rather than relying on manual labeling. The system identifies what needs to be measured and why before constructing justified metric portfolios, outperforming existing baselines on three real-world AI applications including financial and scientific tasks.

Litmus addresses a critical infrastructure gap in AI evaluation: the systematic specification of metrics before implementation. Current approaches assume evaluation targets are already known, forcing teams to retrofit metrics to vague deployment requirements. This research inverts that process by extracting evaluation intent directly from code, enabling metrics to emerge from explicit business logic rather than post-hoc assumptions.

The broader context reflects growing pains in agentic AI systems. As LLM-based applications move from research prototypes into production across finance, healthcare, and other regulated domains, evaluation rigor becomes a competitive differentiator and compliance necessity. Manual metric design doesn't scale when diverse stakeholders have conflicting quality requirements. Litmus's zero-label approach reduces annotation burden while improving metric validity—a significant advantage in real-world deployment where labeled evaluation sets are expensive and slow to generate.

The empirical results carry practical weight. Across financial account grouping, scientific QA, and risk assessment pipelines, Litmus achieved superior concern coverage and validity metrics compared to AutoMetrics and DynamicRubric baselines. The 0.72 Spearman correlation on scientific QA substantially exceeds competing methods' sub-0.47 scores. This demonstrates that metric specification outperforms metric selection when the question shifts from "which metric?" to "what should we measure and why?"

For AI infrastructure companies and enterprise deployers, this work validates automated metric discovery as a viable path toward production-grade evaluation systems. The next phase involves scaling Litmus across heterogeneous pipeline architectures and integrating continuous monitoring frameworks that adapt metrics as system behavior evolves.

• →Litmus uses code analysis and targeted interrogation to automatically specify evaluation metrics without manual labeling
• →The system achieved highest validity on scientific QA evaluation with 0.72 Spearman correlation versus baseline scores below 0.47
• →Zero-label metric specification outperformed traditional automatic metric implementation across three real-world AI pipelines
• →The approach reduces metric redundancy while improving coverage across multiple pipeline stages and evaluation concerns
• →Automated metric specification represents a shift toward production-grade AI evaluation systems that scale with deployment complexity