Quantifying and Optimizing Simplicity via Polynomial Representations

Researchers introduce polynomial representations as a quantitative measure of neural network simplicity, demonstrating that the effective degree of these representations predicts generalization better than existing metrics. The approach yields a differentiable regularizer that improves performance across image classification, text tasks, vision-language models, and reinforcement learning.

This research addresses a fundamental challenge in deep learning: quantifying simplicity in a way that meaningfully predicts how well models generalize to unseen data. While intuition suggests simpler models generalize better, neural networks have lacked a rigorous, practical measure of this property. The polynomial representation framework fills this gap by constructing low-dimensional surrogates of network behavior along data-dependent paths using orthogonal polynomial bases. The effective degree of these polynomials serves as a measurable simplicity metric, outperforming established proxies like loss landscape sharpness.

The work builds on decades of generalization theory but makes it operational for modern deep networks. Traditional measures like weight norms or layer counts fail to capture functional simplicity—what actually matters for learning. By anchoring simplicity to the polynomial degree needed to approximate network outputs, the authors create a distribution-aware metric that naturally accounts for data geometry.

The practical impact extends beyond measurement. The polynomial framework directly enables a differentiable regularizer that consistently improves generalization across diverse domains: vision (image classification, fine-tuning CLIP-style models), natural language processing, and reinforcement learning. This consistency across architectures and tasks suggests the approach captures something fundamental about how neural networks learn.

For practitioners, this offers a principled alternative to ad-hoc regularization techniques. The method provides both interpretability—understanding what makes networks simple—and performance gains. Future work may explore whether polynomial degree becomes a standard metric for model selection and whether tighter theoretical bounds on generalization can be derived from these representations.

• →Polynomial representations provide a quantitative, distribution-aware simplicity metric that outperforms existing generalization proxies like sharpness.
• →Effective polynomial degree predicts generalization consistently across diverse tasks including vision, language, and reinforcement learning.
• →The framework naturally yields a differentiable simplicity regularizer applicable to various domains without task-specific tuning.
• →The approach bridges theory and practice by making the intuitive notion of simplicity-biased learning mathematically precise and actionable.
• →Results suggest polynomial degree could become a standard model selection criterion in deep learning workflows.