Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)
Researchers introduce the Word Coverage Score (WCS), a metric revealing how standard LLM sampling filters (Top-p, Top-k, Min-p) mathematically suppress contextually appropriate vocabulary choices, rendering linguistically valid words unreachable despite existing in the probability space. The study demonstrates that industry-standard decoding defaults unintentionally homogenize text output, acting as hidden censorship mechanisms that limit lexical diversity in generated content.
This research addresses a fundamental tension in modern language model deployment: the gap between latent linguistic capability and actual output diversity. While LLMs possess extensive vocabularies learned during training, the decoding mechanisms that convert probability distributions into text systematically eliminate low-frequency, high-information words that human writers would naturally select. The WCS provides quantitative grounding for observations that LLM-generated text often feels generic despite the models' purported sophistication.
The technical contribution stems from recognizing that sampling filters operate as probability gatekeepers. Top-p nucleus sampling, widely adopted for quality control, accumulates probabilities until reaching a threshold—typically 0.9—then discards remaining candidates regardless of contextual appropriateness. This mechanism evolved to prevent incoherent outputs, but the research demonstrates overcorrection occurs routinely. By measuring the survival rate of legitimate vocabulary across different sampling parameters, the authors create diagnostic tools for practitioners.
For developers and organizations deploying LLMs, this work highlights an optimization frontier previously treated as settled. Current defaults prioritize coherence through aggressive pruning, but the WCS framework enables calibration toward linguistic richness without sacrificing quality. This matters for creative applications, academic writing, and any use case where distinctive voice matters commercially or aesthetically. The research suggests substantial improvements are achievable through parameter tuning rather than architectural changes.
Practitioners should begin auditing their sampling configurations against domain-specific human text corpora to identify suppressed vocabulary patterns. Open-weight model developers specifically gain actionable insights for documentation and default recommendations.
- →Standard LLM sampling filters suppress contextually appropriate vocabulary, rendering valid word choices mathematically unreachable despite existing in probability distributions
- →Word Coverage Score provides quantitative measurement of lexical suppression across decoding parameters, enabling optimization toward diversity without sacrificing coherence
- →Industry-standard sampling defaults prioritize consistency over expressiveness, creating homogenized outputs that underutilize latent model capabilities
- →Practitioners can improve text diversity through sampling parameter tuning rather than model retraining or architectural changes
- →The framework addresses production systems where distinctive voice and linguistic variety have commercial or aesthetic value