AIBullisharXiv – CS AI · Mar 37/104
🧠
Polynomial, trigonometric, and tropical activations
Researchers developed new activation functions for deep neural networks based on polynomial and trigonometric orthonormal bases that can successfully train models like GPT-2 and ConvNeXt. The work addresses gradient problems common with polynomial activations and shows these networks can be interpreted as multivariate polynomial mappings.