Credit-assigned Policy Gradient for Early Stage Retrieval in Two-stage Ranking

Researchers propose Credit-Assigned Policy Gradient (CA-PG), a new machine learning technique that solves the variance problem in training early-stage rankers for two-stage retrieval systems. By computing gradients with respect to individual item selection probability rather than entire candidate sets, CA-PG enables scalable end-to-end training of search and recommendation systems.

This research addresses a fundamental engineering challenge in large-scale information retrieval systems that power search engines, recommendation platforms, and RAG applications. The two-stage ranking architecture is industry-standard because it balances computational efficiency with relevance quality—an initial ranker quickly filters millions of candidates, then a second ranker refines the top results. However, training the early-stage ranker with reinforcement learning has remained computationally intractable.

The core problem stems from variance explosion in vanilla policy gradient methods. When optimizing over candidate set combinations, gradient variance scales exponentially with candidate set size, making training unstable and slow. CA-PG sidesteps this by marginalizing over candidate set composition, focusing instead on whether a specific item appears in any returned set. This mathematical reformulation preserves learning capability while dramatically reducing variance.

The practical implications are significant for infrastructure providers and platform developers. Improved ESR training could enhance ranking quality across search, e-commerce recommendations, and AI-augmented retrieval systems. Faster convergence reduces training compute costs, while better stability enables more sophisticated ranking objectives. The approach works with established models like Plackett-Luce, suggesting broad applicability.

Looking forward, the research opens avenues for more efficient end-to-end optimization of retrieval pipelines. Future work may extend CA-PG to other ranking architectures or apply credit assignment principles to multi-stage systems. The methodology could influence how search platforms and recommendation engines are trained at scale, with potential cost savings and performance gains for companies operating large-scale information retrieval infrastructure.

• →CA-PG reduces gradient variance in early-stage ranker training by marginalizing over candidate set composition rather than processing entire sets
• →The technique enables scalable end-to-end training of retrieval systems critical to search, recommendations, and RAG applications
• →Experiments show improved convergence speed and stability, particularly beneficial for large candidate set sizes typical in production systems
• →The method preserves correctness under reasonably aligned late-stage ranker policies while substantially decreasing training instability
• →Implementation works with canonical Plackett-Luce ranking models, suggesting broad compatibility with existing system architectures