Scaling Retrieval Augmented Generation with RAG Fusion: Lessons from an Industry Deployment

Research on production RAG systems reveals that retrieval fusion techniques like multi-query retrieval and reciprocal rank fusion increase raw document recall but fail to improve end-to-end performance due to re-ranking limits and context constraints. The study found fusion variants actually decreased accuracy from 0.51 to 0.48 while adding latency overhead without corresponding benefits.

• →Retrieval fusion techniques increase raw recall but gains are neutralized after re-ranking and truncation in production environments.
• →Fusion variants failed to outperform single-query baselines, with Hit@10 accuracy decreasing from 0.51 to 0.48.
• →Additional latency overhead from query rewriting and larger candidate sets provides no corresponding effectiveness improvements.
• →Recall-oriented fusion techniques show diminishing returns under realistic production constraints and budgets.
• →The research advocates for evaluation frameworks that jointly consider retrieval quality, system efficiency, and downstream impact.