Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments

Researchers developed a method to improve app store search relevance by using large language models to generate textual relevance judgments, addressing the scarcity of expert-labeled data. A specialized fine-tuned model significantly outperformed general-purpose LLMs in evaluating semantic fit between queries and results.

• →Commercial search systems face challenges with limited expert-provided textual relevance labels compared to abundant behavioral data.
• →A specialized fine-tuned LLM model significantly outperformed general-purpose models for generating textual relevance judgments.
• →The approach combines behavioral relevance (user clicks/downloads) with textual relevance (semantic query fit) to maximize search effectiveness.
• →LLM-generated judgments can help scale search relevance evaluation in commercial applications like app stores.
• →Systematic evaluation of different LLM configurations is crucial for optimal performance in search relevance tasks.