Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods

Researchers propose ADAPT, an online data reweighting framework that dynamically adjusts training sample importance during LLM training rather than using static offline selection methods. This approach maintains data diversity while improving generalization, outperforming existing offline curation techniques on instruction tuning and large-scale pretraining tasks.

The research addresses a fundamental inefficiency in large language model development: data curation remains largely disconnected from the training process itself. Traditional offline methods select or mix training data before training begins, creating brittle pipelines that require complete re-runs when models or tasks change. ADAPT reimagines this problem by treating data curation as a continuous, adaptive process that unfolds alongside model training.

The technical innovation centers on dynamic per-sample reweighting guided by quality signals derived from model similarity metrics. Rather than filtering data and reducing dataset size—a common offline approach that damages diversity—ADAPT maintains the full dataset while adjusting how much each sample influences learning at different training stages. This creates an implicit curriculum that naturally progresses from learning broad patterns to capturing nuanced semantic distinctions.

For the AI development community, this represents meaningful progress toward more efficient and robust training pipelines. Current production workflows often require expensive data curation stages that must be repeated for new domains or model variants. ADAPT's online approach reduces this overhead while demonstrating superior cross-benchmark generalization under equivalent computational budgets. The framework shows consistent improvements across both instruction tuning and pretraining scenarios, suggesting broad applicability.

The implications extend to model development economics. If online reweighting achieves better results with the same compute budget, teams can either reduce training costs or allocate resources toward larger models. This could accelerate the pace of LLM improvements, particularly for resource-constrained organizations. The technique's ability to adapt dynamically also suggests better performance on specialized domains without separate curation pipelines, democratizing high-quality model development.

• →ADAPT replaces static offline data curation with dynamic online reweighting, reducing engineering overhead and brittleness.
• →The method maintains full dataset diversity while adjusting sample importance during training via adaptive per-sample learning rates.
• →Online reweighting achieves stronger cross-benchmark generalization than offline selection/mixing under equal computational budgets.
• →ADAPT functions as an implicit curriculum learner, progressively shifting from coarse patterns to fine-grained semantic distinctions.
• →The framework eliminates the need to re-run entire curation pipelines when encountering model or task distribution shifts.