Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds

Researchers introduce intrinsic Muon (iMuon), a unified optimization framework that extends the Muon optimizer to Riemannian manifolds while preserving symmetries and enabling closed-form solutions. The approach demonstrates applications in LLM fine-tuning, image classification, and subspace learning with convergence guarantees dependent only on manifold dimension rather than factor conditioning.

The development of iMuon addresses a fundamental limitation in modern optimization algorithms used for large-scale machine learning. The Muon optimizer has gained prominence for its effectiveness in training large models, but its application to constrained parameter spaces—such as low-rank factorizations and orthogonal matrices—remained mathematically intractable. This work resolves that gap by leveraging intrinsic geometric properties of Riemannian manifolds.

The key innovation involves recognizing that unitarily invariant Euclidean norms naturally lift to intrinsic norms on manifold tangent spaces, preserving the symmetries that naive tangent-space restrictions would break. This theoretical insight yields practical algorithms with closed-form updates across multiple important manifold classes, including fixed-rank and symmetric positive definite matrices.

For practitioners, the implications are substantial. In LoRA fine-tuning of large language models, the framework's rank-dependent convergence rate—independent of factor conditioning—eliminates computational bottlenecks that plagued prior methods requiring manual rescaling. The rate guarantees depend only on manifold dimension rather than numerical properties of the factorization, enabling more predictable training dynamics.

The experimental validation across diverse domains signals that iMuon could become foundational infrastructure in the optimization toolkit for modern deep learning. Its ability to handle multiple manifold geometries within a unified framework reduces implementation complexity while improving convergence behavior. This work represents incremental but meaningful progress in making advanced optimization methods more accessible and efficient for real-world large-scale learning problems.

• →iMuon extends the Muon optimizer to Riemannian manifolds while preserving geometric symmetries through intrinsic norm constraints.
• →Convergence rates depend only on manifold dimension, eliminating dependence on factor conditioning in fixed-rank settings.
• →Framework provides closed-form solutions on fixed-rank, SPD, Stiefel, and Grassmann manifolds with any unitarily invariant norm.
• →LoRA fine-tuning experiments demonstrate practical efficiency gains by removing prior runtime rescaling requirements.
• →Unified approach reduces implementation complexity for manifold-constrained optimization across diverse machine learning applications.