Can Small Training Runs Reliably Guide Data Curation? Rethinking Proxy-Model Practice

Researchers demonstrate that small-scale proxy models commonly used by AI companies to evaluate data curation strategies produce unreliable conclusions because optimal training configurations are data-dependent. They propose using reduced learning rates in proxy model training as a simple, cost-effective solution that better predicts full-scale model performance across diverse data recipes.

The research addresses a fundamental disconnect in how frontier AI labs develop training data strategies. Currently, teams train small proxy models with fixed hyperparameters across different data recipes to maintain experimental fairness, then apply insights to massive production models. This study reveals a critical flaw: conclusions about data quality rankings flip when hyperparameters shift, because each dataset reaches optimal performance under different training configurations. This misalignment matters because full-scale production pipelines always include hyperparameter optimization, making the fixed-configuration protocol fundamentally incompatible with real-world practice.

The finding has immediate practical implications for AI development efficiency. Hyperparameter tuning small models for each data recipe variant is computationally expensive, creating pressure to use shortcuts that produce misleading results. The proposed solution—training proxy models with reduced learning rates—elegantly sidesteps this cost problem. The approach theoretically preserves dataset quality rankings while requiring minimal additional computation, validated across 23 different data curation recipes spanning critical dimensions like source quality, diversity, and preprocessing methods.

For AI companies, this research translates to potentially significant cost savings in data pipeline optimization. Currently, misjudgments from unreliable proxy experiments can cascade into months of wasted compute on suboptimal pretraining runs for massive models. Better small-scale prediction methodology reduces exploration costs and accelerates time-to-optimal-model. The work also has broader implications for AI reproducibility and scientific rigor, establishing clearer protocols for data evaluation that the community can standardize on moving forward.

• →Small proxy models with fixed hyperparameters produce unreliable data recipe rankings because optimal configurations vary per dataset
• →The standard evaluation protocol diverges from production pipelines where hyperparameter optimization is routine practice
• →Reduced learning rates in proxy training correlate strongly with full-scale LLM performance without excessive computational overhead
• →The fix was validated across 23 data recipes spanning four critical dimensions of data curation methodology
• →Better proxy evaluation protocols could reduce wasted compute on suboptimal pretraining runs for frontier AI models