When Data Is Scarce: Scaling Sparse Language Models with Repeated Training

Researchers demonstrate that sparse neural networks can improve scaling efficiency in data-limited training scenarios, where models must train multiple epochs on repeated data. The study introduces a scaling law predicting performance across varying sparsity levels (up to 93.75%), finding that moderate sparsity around 50% optimizes loss while higher sparsity improves compute efficiency, challenging assumptions that sparsity is purely an efficiency tool.

This research addresses a critical gap in large language model optimization: how sparsity behaves when data constraints force repeated training cycles. Most scaling law research assumes infinite unique data, leaving practitioners without guidance for real-world scenarios where data budgets are finite. The study's comprehensive experiments—spanning up to 1.92B parameter models with 16 training epochs—provide empirical evidence that sparsity fundamentally alters how networks respond to data repetition.

The findings reveal a counterintuitive mechanism: sparse models delay the onset of diminishing returns from repeated data, making multi-epoch training substantially more effective than dense equivalents. This has profound implications for training efficiency. Rather than viewing sparsity as purely a post-hoc compression technique, the research frames it as an architectural lever that trades off loss-optimal performance against compute-optimal efficiency depending on specific resource constraints.

For AI infrastructure providers and practitioners, this shifts the calculus around model design choices. Organizations with limited unique data can now leverage sparsity strategically during training rather than only at inference. The practical impact extends to edge deployment, where both training and inference efficiency matter. The clear quantification of resource trade-offs—moderate sparsity for loss optimization versus aggressive sparsity for compute gains—enables more informed architectural decisions.

The open-sourced code democratizes implementation, allowing developers to apply these scaling laws to their specific data and compute budgets. Future work should explore how these findings interact with other efficiency techniques like quantization or distillation, and whether the observed delayed saturation effect extends to even larger models.

• →Sparse models delay diminishing returns from repeated data, making multi-epoch training more effective in data-constrained scenarios.
• →Optimal sparsity levels differ by objective: ~50% for loss minimization versus higher levels for compute efficiency.
• →A new scaling law accurately predicts sparse model performance across varying active parameters, data budgets, and repetition counts.
• →Sparsity functions as a mechanism for improving scaling trade-offs rather than merely an inference efficiency tool.
• →Findings validated across models from 1.92B to 7.68B parameters with up to 41.6B total training tokens.