#expressivity News & Analysis

3 articles tagged with #expressivity. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AIBullisharXiv – CS AI · Jun 257/10

🧠

Rational Neural Networks have Expressivity Advantages

Researchers demonstrate that neural networks using trainable rational activation functions achieve exponentially better parameter efficiency and expressivity compared to standard activations like ReLU, Sigmoid, and Tanh. The findings show rational activations require only polylogarithmic overhead to approximate fixed-activation networks, while the reverse requires logarithmic parameters—a theoretical advantage that translates to practical performance gains.

AINeutralarXiv – CS AI · Jun 56/10

🧠

Separation Power of Equivariant Neural Networks

Researchers characterize the separation power of equivariant neural networks, demonstrating that non-polynomial activations like ReLU and sigmoid achieve equivalent maximum expressivity, while depth and architectural choices significantly influence a model's ability to distinguish inputs. This theoretical analysis provides a framework for comparing model expressivity and understanding the design principles behind convolutional and permutation-invariant networks.

AIBullisharXiv – CS AI · May 276/10

🧠

More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

Researchers propose Mixture of Activations (MoA), a novel feedforward network design that dynamically selects activation functions per token rather than applying a single fixed function across all inputs. Theoretical analysis proves MoA offers strict expressivity advantages over fixed-activation networks, while empirical testing on language models up to 2B parameters demonstrates consistent improvements in loss metrics with minimal computational overhead.