Scaling with Collapse: Efficient and Predictable Training of LLM Families

Researchers demonstrate that training loss curves for large language models can collapse onto universal trajectories when hyperparameters are optimally set, enabling more efficient LLM training. They introduce Celerity, a competitive LLM family developed using these insights, and show that deviation from collapse can serve as an early diagnostic for training issues.

• →LLM training loss curves collapse onto universal trajectories when optimization hyperparameters are set optimally for the data budget.
• →Loss curve collapse serves as a signature of compute-efficient training across different model scales.
• →Deviation from collapse provides early detection of training pathologies before significant compute is wasted.
• →The predictability of collapsed curves enables more efficient early stopping in hyperparameter tuning.
• →The research team successfully trained Celerity, a competitive LLM family using these collapse-based insights.