Muon Learns More Robust and Transferable Features than Adam

Research demonstrates that Muon, an emerging optimizer for large language models and vision classifiers, produces more robust and transferable features than Adam and SGD across multiple architectures. The study shows Muon-learned features maintain superior performance on corrupted data and transfer more effectively to downstream tasks, with theoretical support provided through margin and effective rank analysis.

Muon's emergence as a state-of-the-art optimizer represents a meaningful advancement in how neural networks learn generalizable representations. This research moves beyond simple efficiency comparisons to examine the quality of learned features, revealing that Muon produces representations with superior robustness to data corruption and stronger transferability properties—critical factors for practical deployment of AI systems.

The significance lies in understanding why Muon outperforms established optimizers like Adam. By measuring logit margins and effective rank across network layers, researchers establish that Muon achieves larger decision boundaries and greater diversity in hidden state representations. These properties directly correlate with real-world performance advantages: models trained with Muon demonstrate consistent robustness improvements when evaluated on corrupted images and texts, and their learned features adapt more effectively when applied to new tasks, whether through linear probing or full fine-tuning.

For the machine learning community, this work has substantial implications for training pipelines. Organizations investing in large-scale model pretraining face trade-offs between computational efficiency and feature quality. Muon's demonstrated advantages in both dimensions challenge the dominance of Adam in production systems. The theoretical analysis provides principled explanations for empirical observations, suggesting Muon's benefits are structural rather than incidental.

Looking forward, the research invites investigation into whether Muon's advantages extend to other domains and model scales, and whether its properties can be incorporated into hybrid approaches. The work also raises questions about why existing optimizers like Adam haven't converged on similar feature-learning characteristics, potentially revealing fundamental insights about optimization landscapes for deep learning systems.

• →Muon optimizer produces features that are consistently more robust to data corruption than Adam and SGD across transformers and CNNs
• →Muon-learned representations transfer more effectively to downstream tasks with superior linear probe and fine-tuning performance
• →Theoretical analysis proves Muon achieves larger margins and higher effective rank in multi-component classification problems
• →Feature robustness correlates with larger logit margins across network layers, providing mechanistic insight into Muon's advantages
• →Results suggest Muon should be reconsidered for production pretraining pipelines where both efficiency and feature quality matter