Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments

Apple's App Store search team successfully implemented LLM-generated textual relevance labels to augment their ranking system, addressing data scarcity issues. A fine-tuned specialized model outperformed larger pre-trained models, generating millions of labels that improved search relevance. This resulted in a statistically significant 0.24% increase in conversion rates in worldwide A/B testing.

• →Fine-tuned specialized LLMs significantly outperformed larger pre-trained models for generating textual relevance labels.
• →LLM-generated labels successfully addressed the scarcity of expert-provided textual relevance data at scale.
• →Augmenting behavioral relevance with textual relevance improved both offline metrics and real-world performance.
• →A worldwide A/B test on App Store search showed a statistically significant 0.24% conversion rate increase.
• →The biggest performance gains occurred in tail queries where behavioral data is typically unreliable.