#numerical-stability News & Analysis

2 articles tagged with #numerical-stability. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

2 articles

AIBearisharXiv – CS AI · Apr 207/10

🧠

The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference

Researchers have discovered that FP16 floating-point precision causes systematic numerical divergence between KV-cached and cache-free inference in transformer models, producing 100% token divergence across multiple architectures. This challenges the long-held assumption that KV caching is numerically equivalent to standard computation, with controlled FP32 experiments confirming FP16 non-associativity as the causal mechanism.

AINeutralarXiv – CS AI · 3d ago6/10

🧠

FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo

Researchers propose FOAM, an adaptive algorithm that addresses the computational bottleneck in Shampoo optimization by dynamically controlling damping factors and eigendecomposition frequency to mitigate errors from stale preconditioner updates. The method reduces wall-clock training time while maintaining convergence stability, offering a practical solution to the efficiency-fidelity trade-off in large-scale machine learning optimization.