AINeutralarXiv – CS AI · 14h ago6/10
🧠
Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
Researchers introduce Singularity-aware Adam (S-Adam), a novel optimizer addressing instability in deep learning with non-smooth components like ReLU activations. The method uses a Local Geometric Instability metric to dynamically adjust step sizes, demonstrating up to 6% accuracy improvements on benchmark datasets while mitigating gradient oscillations.