Paraphrase Brittleness in Production Retrieval-Augmented Commercial Recommendation: Reproducibility Below the Rerun-Stability Baseline

Research reveals that AI recommendation systems exhibit severe brittleness when processing paraphrased queries, with recommendation-set similarity dropping to 0.288 for cosmetic rewordings and 0.135 for constraint-modified queries—far below the 0.50-0.61 baseline for identical prompts. This undermines the reliability of AI visibility tracking metrics used in commercial recommendation optimization, as brand mention frequency depends more on prompt phrasing than actual model behavior.

A comprehensive study of OpenAI and Anthropic models demonstrates fundamental instability in how AI assistants recommend brands based on subtle query variations. When researchers tested approximately 6,000 paraphrase runs against 6,000 identical-prompt controls, they found that cosmetic rewording changes—such as "best CRM" versus "top CRM"—produced recommendation sets with only 14-29% overlap, compared to 50-61% overlap for exact prompt repetition. This variance increases dramatically when queries include additional constraints like geographic region or specificity requirements.

The research exposes a critical flaw in emerging AI Expansion Optimization and AI-driven Search Engine Optimization practices, which track brand "AI visibility" by monitoring mention frequencies across fixed prompt sets. These tracking methodologies assume consistency in recommendation behavior, but the dominant source of variance is not model stability—it is the arbitrary choice of phrasing used by the tracker. Even increasing reasoning effort through higher computational parameters fails to narrow the gap meaningfully, with improvements bounded at ±0.05.

For the AI industry and commercial stakeholders relying on AI recommendation systems, this finding suggests that current measurement frameworks are fundamentally unreliable. Brand visibility metrics based on prompt-by-prompt mention tracking collapse as meaningful units of measurement because they capture paraphrase artifacts rather than genuine model behavior. The natural space of buyer-phrasing variations vastly exceeds the scale of benchmark prompt sets that academic literature has validated multi-prompt evaluation methods against. Resolving this brittleness likely requires architectural changes to how AI systems process intent rather than simply expanding prompt coverage.

• →AI recommendation systems show 14-29% overlap in brand suggestions when identical queries are rephrased, versus 50-61% overlap for exact prompt repetition, indicating severe brittleness.
• →Current AI visibility tracking metrics are structurally unstable because variance primarily stems from prompt phrasing choice rather than actual model behavior toward brands.
• →Increasing computational reasoning effort does not meaningfully reduce paraphrase sensitivity, limiting the effectiveness of resource-intensive optimization approaches.
• →The natural space of buyer-phrasing variations far exceeds validated scales for multi-prompt evaluation methods, making conventional prompt-set expansion insufficient.
• →Addressing this brittleness likely requires fundamental changes to intent-processing architecture rather than incremental improvements to measurement methodology.