#numerical-stability News & Analysis

3 articles tagged with #numerical-stability. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AIBearisharXiv – CS AI · Apr 207/10

🧠

The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference

Researchers have discovered that FP16 floating-point precision causes systematic numerical divergence between KV-cached and cache-free inference in transformer models, producing 100% token divergence across multiple architectures. This challenges the long-held assumption that KV caching is numerically equivalent to standard computation, with controlled FP32 experiments confirming FP16 non-associativity as the causal mechanism.

AINeutralarXiv – CS AI · Jun 56/10

🧠

PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training

Researchers propose a PC (Preconditioning) layer that uses polynomial weight parameterization to stabilize training of large language models while maintaining computational efficiency. The approach demonstrates performance improvements over standard transformers during Llama-1B pre-training and includes theoretical guarantees for convergence in certain network architectures.

🧠 Llama

AINeutralarXiv – CS AI · Jun 26/10

🧠

FOAM: Frequency and Operator Error-Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo

Researchers propose FOAM, an adaptive algorithm that addresses the computational bottleneck in Shampoo optimization by dynamically controlling damping factors and eigendecomposition frequency to mitigate errors from stale preconditioner updates. The method reduces wall-clock training time while maintaining convergence stability, offering a practical solution to the efficiency-fidelity trade-off in large-scale machine learning optimization.