2 articles tagged with #loss-geometry. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers demonstrate that worker disagreement in Local SGD training reveals the underlying loss geometry of deep neural networks, providing a computationally efficient method to estimate dominant Hessian directions without expensive direct calculations. This finding has implications for optimizing distributed training of large models like Transformers.
Researchers derive a closed-form upper bound for the Hessian eigenspectrum of cross-entropy loss in smooth nonlinear neural networks using the Wolkowicz-Styan bound. This analytical approach avoids numerical computation and expresses loss sharpness as a function of network parameters, training sample orthogonality, and layer dimensions—advancing theoretical understanding of the relationship between loss geometry and generalization.
