How AI training scales

Researchers discovered that gradient noise scale can predict how well neural network training parallelizes across different tasks. This finding suggests that larger batch sizes will become increasingly useful for complex AI training, potentially removing scalability limits for future AI systems.