SAGE: Scalable AI Governance & Evaluation

Researchers and LinkedIn introduce SAGE, a framework that combines human judgment with AI surrogates to evaluate search relevance at scale. By using a bidirectional calibration loop between policy, precedent examples, and LLM judges, the system achieves near-human agreement while reducing inference costs by 92×, ultimately driving a 0.25% lift in LinkedIn's daily active users.

SAGE addresses a critical operational challenge in modern search systems: the inability to scale high-quality human evaluation across millions of queries without prohibitive costs. Traditional engagement metrics often miss relevance failures that matter most to users, creating a blind spot in production systems. LinkedIn's solution elegantly bridges this gap by using an LLM surrogate judge trained through iterative calibration with human annotators, effectively democratizing expert-level judgment across the platform's search ecosystem.

The framework's innovation lies in its bidirectional feedback mechanism. Rather than treating human judgment as static ground truth, SAGE treats policy, precedents, and the surrogate model as co-evolving components that refine each other. Natural language policies define relevance criteria, accumulated precedents provide concrete examples, and the LLM learns to apply these consistently. This approach transforms inherently subjective relevance judgments into reproducible, multi-dimensional rubrics with measurable inter-annotator agreement.

The deployment impact validates the theoretical benefits. By applying teacher-student distillation, LinkedIn compressed expensive frontier model reasoning into lightweight student models suitable for production inference, achieving 92× cost reduction while maintaining fidelity. The system detected regressions invisible to engagement metrics—a critical capability since engagement proxies can mislead when relevance and engagement diverge.

The 0.25% lift in daily active users carries significant financial implications for LinkedIn's parent company Microsoft. This suggests that search quality improvements directly influence user retention and platform engagement. For the AI industry, SAGE demonstrates that combining human oversight with intelligent automation can scale governance without sacrificing judgment quality, offering a template for other platforms managing similar evaluation challenges at massive scale.

• →SAGE uses bidirectional calibration between human policy, precedent examples, and LLM judges to scale relevance evaluation without proportional cost increases.
• →Teacher-student distillation reduced inference costs by 92× while preserving judgment quality for production deployment.
• →The framework detected relevance regressions invisible to traditional engagement metrics, proving that structured AI evaluation can outperform proxy-based methods.
• →LinkedIn's deployment achieved 0.25% lift in daily active users, demonstrating measurable business impact from improved search relevance governance.
• →SAGE's approach of treating human judgment and AI surrogates as co-evolving components offers a replicable pattern for large-scale AI system evaluation.