FP8 is All You Need (Part 1): Debunking Hardware FP64 as the HPC Holy Grail

A research paper challenges the long-held belief that native FP64 (double-precision) hardware is essential for scientific computing, arguing that FP8 tensor operations combined with advanced mathematical schemes can achieve equivalent accuracy at dramatically higher speeds on modern GPUs like NVIDIA's Blackwell B300.

This academic paper fundamentally challenges decades of HPC orthodoxy by demonstrating that low-precision arithmetic, when combined with sophisticated mathematical reconstruction techniques, can replace native hardware double-precision without sacrificing accuracy. The research centers on NVIDIA's Blackwell B300 GPU, which paradoxically crippled native FP64 performance to just 1.3 TFLOPS—a catastrophic 31x regression compared to its predecessor. Rather than viewing this as a limitation, the authors reframe it as an opportunity to leverage abundant FP8 tensor throughput through the Ozaki Scheme II, a Chinese Remainder Theorem-based reconstruction algorithm that recovers full FP64 accuracy while achieving roughly 500 TFLOPS—a 385x improvement over native performance. The implications are profound for high-performance computing workloads spanning sparse matrix multiplication, general matrix-vector operations, and stencil computations. The research introduces the Tensor-Memory Equilibrium model to analyze where computation becomes memory-bound, revealing that register-level fusion effectively eliminates the overhead of emulation. For GPU manufacturers, this signals a strategic pivot away from expensive native FP64 silicon toward tensor-optimized architectures, fundamentally altering hardware design priorities. For HPC practitioners, the findings suggest substantial performance gains and cost efficiency improvements are achievable through algorithmic innovation rather than raw silicon capability. The work positions low-precision arithmetic as the future of scientific computing, contingent on algorithmic sophistication.

• →FP8 tensor operations with Ozaki Scheme II reconstruction can achieve full FP64 accuracy while delivering 500 TFLOPS on B300 GPUs, versus 1.3 TFLOPS native.
• →NVIDIA's Blackwell B300 intentionally reduced native FP64 performance, making low-precision emulation approaches strategically advantageous.
• →Register-level fusion makes FP8 emulation essentially free behind the memory wall, eliminating performance overhead of mathematical reconstruction.
• →The Tensor-Memory Equilibrium model provides new analytical framework for GPU performance optimization beyond traditional roofline analysis.
• →Native FP64 hardware is becoming obsolete for HPC workloads as algorithmic techniques prove superior to raw silicon capabilities.