Researchers demonstrate that training loss curves for large language models can collapse onto universal trajectories when hyperparameters are optimally set, enabling more efficient LLM training. They introduce Celerity, a competitive LLM family developed using these insights, and show that deviation from collapse can serve as an early diagnostic for training issues.
Researchers developed a new approach to quantization-aware training (QAT) that optimizes compute allocation between full-precision and quantized training phases. They discovered that contrary to previous findings, the optimal ratio of QAT to full-precision training increases with total compute budget, and derived scaling laws to predict optimal configurations across different model sizes and bit widths.
A new analysis reveals that compute requirements for training neural networks to match ImageNet classification performance have decreased by 50% every 16 months since 2012. Training a network to AlexNet-level performance now requires 44 times less compute than in 2012, far outpacing Moore's Law improvements which would only yield 11x cost reduction over the same period.
