The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval

Researchers discovered that Large Language Models exhibit a U-shaped performance degradation curve when processing text with word-boundary corruption, termed the 'Text Uncanny Valley.' This reveals a critical vulnerability in LLM robustness: performance worsens at moderate corruption levels before improving again at extreme corruption, suggesting models struggle during transitions between word-level and character-level processing modes.

This research exposes a fundamental blind spot in LLM evaluation methodologies. Current benchmarking focuses on clean, syntactically correct inputs, creating a false confidence in model robustness. The study demonstrates that moderate text corruption—such as inserting whitespace within words—triggers worse performance than both minimally corrupted and heavily fragmented text, a counterintuitive finding with serious implications for real-world deployment.

The mode transition hypothesis offers compelling mechanistic insight. LLMs operate effectively in specialized modes: word-level processing for near-normal text and character-level reconstruction for heavily fragmented text. However, at intermediate corruption levels, models oscillate between these modes ineffectively, creating a performance valley. This explains why in-context learning fails to bridge the gap and why regularized perturbations substantially reduce the U-shape effect.

For practitioners deploying LLMs in production environments involving noisy, uncurated, or user-generated text—common in social media analysis, web scraping, or real-time data ingestion—this research signals potential brittleness. The effect appears task-dependent; math reasoning shows the U-shape in weaker models but not in stronger ones, suggesting that higher-capacity or better-trained models mitigate this failure mode more effectively.

The tokenization entropy analysis strengthens the interpretation, with peak entropy preceding minimum F1 scores. This indicates the valley represents genuine computational confusion rather than statistical noise. Future robustness research must move beyond clean-text paradigms and systematically evaluate performance across corruption regimes. Organizations relying on LLMs should test against naturally occurring text degradation patterns before deployment.

• →LLMs show U-shaped performance degradation under moderate text corruption, creating an 'uncanny valley' invisible to standard benchmarks
• →The effect stems from models transitioning between word-level and character-level processing modes, with intermediate corruption preventing effective operation in either mode
• →In-context learning cannot rescue performance in the valley, but regularized perturbations substantially reduce the U-shaped curve
• →The failure mode is less pronounced in stronger models and tasks requiring less exact lexical matching, suggesting architectural or training improvements can mitigate the issue
• →Real-world deployments processing noisy or uncurated text need evaluation protocols beyond clean-text benchmarks to assess actual robustness