Answer Presence Drives RAG Rewriting Gains

A new research audit challenges the assumed benefits of LLM rewriters in retrieval-augmented QA systems, finding that performance gains stem primarily from the presence of gold answer strings in rewritten context rather than from genuine passage curation. The study introduces controlled intervention methods to test rewriter claims, revealing that conventional evaluation probes are sensitive to methodology choices and may report misleading results.

This research addresses a critical gap in understanding how retrieval-augmented generation (RAG) systems actually work. Developers have long assumed that LLM rewriters improve QA performance by curating and enhancing retrieved passages—a reasonable hypothesis that has driven adoption of these multi-stage architectures. However, this audit demonstrates that the primary driver of F1 improvements is simpler: whether the correct answer appears in the rewritten context at all.

The controlled intervention methodology is rigorous. By systematically removing answer spans, injecting them into rewrites that lacked them, and comparing against placebo edits, researchers isolated the causal factor with precision. Results across multiple reader models, datasets, and configurations show remarkably consistent patterns: F1 drops of 28-64 points when answers are removed, and gains of 0.7-9.7 points when answers are injected into previously answer-free rewrites. This suggests that rewriters function less as quality-enhancement tools and more as answer-locators or amplifiers.

Equally important is the critique of evaluation methodology itself. The study's sentinel audit reveals that standard single-mask probing produces unreliable results that flip sign and fail statistical equivalence tests under modest variations. This methodological fragility undermines confidence in previously published rewriter-gain claims across the field.

For practitioners, the implications are substantial. Organizations investing in complex rewriting pipelines should question whether simpler answer-injection or answer-highlighting approaches might achieve similar gains at lower computational cost. For researchers, the work establishes new evaluation standards that prevent misleading claims about architectural improvements. The released intervention runner and sentinel panel enable systematic, reproducible testing of future rewriter proposals.

• →LLM rewriter gains in RAG systems are primarily driven by answer string presence, not passage curation quality
• →Removing correct answers from rewritten context causes 28-64 point F1 drops, establishing causal mechanism through controlled intervention
• →Conventional single-mask evaluation probes produce unreliable results that flip under alternative sentinel configurations
• →Rewriter performance claims require robust methodology testing against multiple evaluation approaches to avoid misleading conclusions
• →Simpler answer-injection mechanisms may replicate rewriter benefits at lower computational cost