HiFloat4 Format for Language Model Pre-training on Ascend NPUs

Researchers demonstrate that HiFloat4, a 4-bit floating-point format, enables efficient large language model training on Huawei's Ascend NPUs with up to 4x improvements in compute throughput and memory efficiency. The study shows that specialized stabilization techniques can maintain accuracy within 1% of full-precision baselines while preserving computational gains across dense and mixture-of-experts architectures.

This research addresses a critical challenge in AI infrastructure: reducing the computational and memory overhead of training large foundation models without sacrificing quality. By leveraging 4-bit floating-point formats specifically optimized for Huawei's Ascend NPUs, the work demonstrates that extreme quantization remains viable at scale, with careful numerical stabilization techniques preventing the accuracy degradation typically associated with such aggressive compression.

The significance extends beyond pure engineering optimization. As large language models grow increasingly expensive to train, hardware manufacturers and researchers must develop format innovations that unlock efficiency gains on proprietary accelerators. Huawei's Ascend NPU ecosystem has gained prominence as geopolitical tensions drive technology decoupling, making optimized training techniques for these chips strategically important. The comparison between HiFloat4 and MXFP4 formats provides empirical grounding for architecture-specific format choices.

For the AI infrastructure market, this work validates that 4-bit training techniques scale successfully to both dense and mixture-of-experts models—two dominant architectural paradigms in modern LLM development. Maintaining sub-1% accuracy loss while achieving 4x throughput improvements directly impacts training costs and iteration speed, metrics that determine competitive advantage in model development. Organizations training models on Ascend infrastructure gain quantifiable performance improvements.

Looking forward, the continued refinement of low-precision training formats across competing hardware platforms (NVIDIA, Huawei, others) will reshape infrastructure decisions. The focus shifts from whether extreme quantization works to which hardware-format combinations deliver optimal cost-performance ratios for specific use cases.

• →HiFloat4 format achieves 4x improvements in compute throughput and memory efficiency on Ascend NPUs compared to higher-precision baselines
• →Stabilization techniques maintain relative error within 1% of full-precision models while preserving 4-bit computational efficiency
• →Research validates FP4 training effectiveness across both dense architectures and mixture-of-experts models at large scale
• →Format optimization directly impacts training costs and iteration speed for organizations using Huawei Ascend infrastructure
• →Study provides empirical comparison between HiFloat4 and MXFP4 formats for guiding hardware-specific architectural decisions