AIBullisharXiv – CS AI · 6h ago7/10
🧠
Rational Neural Networks have Expressivity Advantages
Researchers demonstrate that neural networks using trainable rational activation functions achieve exponentially better parameter efficiency and expressivity compared to standard activations like ReLU, Sigmoid, and Tanh. The findings show rational activations require only polylogarithmic overhead to approximate fixed-activation networks, while the reverse requires logarithmic parameters—a theoretical advantage that translates to practical performance gains.