Not All Queries Need Rewriting: When Prompt-Only LLM Refinement Helps and Hurts Dense Retrieval

Research reveals that LLM query rewriting in RAG systems shows highly domain-dependent performance, degrading retrieval effectiveness by 9% in financial domains while improving it by 5.1% in scientific contexts. The study identifies that effectiveness depends on whether rewriting improves or worsens lexical alignment between queries and domain-specific terminology.

• →LLM query rewriting performance varies dramatically across domains, with 9% degradation in finance and 5.1% improvement in scientific retrieval tasks.
• →Rewriting degrades performance when it replaces domain-specific terms in already well-matched queries, reducing lexical alignment.
• →Improvements occur when rewriting shifts queries toward corpus-preferred terminology and resolves inconsistent nomenclature.
• →95% of all rewrites involve lexical substitution, with effectiveness dependent on the direction rather than presence of substitution.
• →Domain-adaptive post-training is recommended as a safer strategy than prompt-only rewriting in well-optimized verticals.