AIBullisharXiv – CS AI · 6h ago7/10
🧠
Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
Researchers identify a fundamental flaw in current FP4 training approaches for large language models: E2M1 formats suffer from systematic "Shrinkage Bias" that degrades training stability. They propose UFP4, a uniform 4-bit recipe using E1M2/INT4 grids that outperforms existing E2M1 baselines across multiple model scales, suggesting future AI accelerators should prioritize uniform grid formats for training.
🏢 Nvidia