Cram Less to Fit More: Training Data Pruning Improves Memorization of Facts

Researchers present a data pruning technique that improves how large language models memorize factual knowledge by optimizing training data distribution. The work, grounded in information-theoretic analysis, addresses the gap between theoretical model capacity and actual factual accuracy, offering practical methods to reduce hallucinations in knowledge-intensive tasks.

This research tackles a fundamental challenge in large language model deployment: the persistent gap between model capacity and factual accuracy. Rather than scaling model size, the authors take a data-centric approach, demonstrating that training data distribution significantly impacts how effectively models retain factual information. By pruning redundant or suboptimal training examples, they show measurable improvements in fact memorization without architectural changes.

The problem context is critical for understanding impact. LLMs increasingly power production systems requiring high factual accuracy—from search to enterprise knowledge bases—yet hallucinations remain a primary failure mode. Previous solutions focused on model scaling, fine-tuning, or retrieval augmentation. This work shifts perspective by revealing that simply including more data doesn't guarantee better fact retention; information-theoretic saturation points exist beyond which additional training examples degrade performance.

For practitioners and organizations, this finding carries direct implications. Data pruning offers a cost-efficient alternative to model scaling or complex retrieval systems. Smaller, better-trained models become competitive with larger ones, reducing computational overhead and inference costs. This particularly benefits resource-constrained organizations building domain-specific systems where factual accuracy is non-negotiable.

The research opens several investigation paths: optimal pruning algorithms for different fact distributions, scaling effects across model sizes, and applicability to multimodal systems. As foundation models transition from research curiosities to production infrastructure, understanding data efficiency becomes as critical as model architecture. Future work likely combines these pruning insights with active learning strategies to further minimize training requirements while maximizing factual fidelity.

• →Training data pruning can improve factual memorization in LLMs without increasing model size or computational cost
• →Information-theoretic analysis reveals optimal training data distributions contain saturation points where additional examples harm accuracy
• →Data-centric approaches offer cost-efficient alternatives to model scaling for reducing hallucinations in knowledge-intensive tasks
• →Smaller, better-pruned models could outperform larger models on factual tasks, enabling practical deployment in resource-constrained environments
• →Findings suggest future LLM development should prioritize data quality and distribution optimization alongside architectural improvements