AIBullisharXiv – CS AI · Apr 207/10
🧠
StoSignSGD: Unbiased Structural Stochasticity Fixes SignSGD for Training Large Language Models
Researchers introduce StoSignSGD, a novel optimization algorithm that fixes convergence issues in SignSGD by injecting structural stochasticity while maintaining unbiased updates. The algorithm demonstrates 1.44x to 2.14x speedup in low-precision FP8 LLM pretraining where AdamW fails, and outperforms existing optimizers in mathematical reasoning fine-tuning tasks.
