AIBullisharXiv – CS AI · Jun 257/10
🧠Researchers demonstrate that neural networks using trainable rational activation functions achieve exponentially better parameter efficiency and expressivity compared to standard activations like ReLU, Sigmoid, and Tanh. The findings show rational activations require only polylogarithmic overhead to approximate fixed-activation networks, while the reverse requires logarithmic parameters—a theoretical advantage that translates to practical performance gains.
AIBullisharXiv – CS AI · Jun 97/10
🧠Researchers introduce I-Segmenter, the first fully integer-only Vision Transformer framework for semantic segmentation that eliminates floating-point operations to enable efficient deployment on resource-constrained devices. The model achieves only 5.1% accuracy loss compared to standard floating-point versions while reducing model size by 3.8x and improving inference speed by 1.2x, with a novel activation function addressing quantization challenges.
AIBullisharXiv – CS AI · Mar 37/104
🧠Researchers developed new activation functions for deep neural networks based on polynomial and trigonometric orthonormal bases that can successfully train models like GPT-2 and ConvNeXt. The work addresses gradient problems common with polynomial activations and shows these networks can be interpreted as multivariate polynomial mappings.
AINeutralarXiv – CS AI · Jun 95/10
🧠SmartMixed introduces a two-phase training strategy enabling neural networks to learn optimal per-neuron activation functions dynamically, then fix them for efficient inference. The approach allows different neurons to select from six candidate activation functions based on learned preferences, demonstrating that layer-specific activation choices improve network performance compared to uniform activation function architectures.
AINeutralarXiv – CS AI · Jun 26/10
🧠Researchers discovered that continuous-time RNNs trained with noise injected inside activation functions paradoxically perform best when noise remains present at test time, contradicting conventional assumptions about noise removal. This phenomenon stems from noise-induced shifts in neural network dynamics that become computationally integrated into learned representations, revealing that networks can overfit to training noise itself rather than just input-output mappings.
AIBullisharXiv – CS AI · May 276/10
🧠Researchers propose Mixture of Activations (MoA), a novel feedforward network design that dynamically selects activation functions per token rather than applying a single fixed function across all inputs. Theoretical analysis proves MoA offers strict expressivity advantages over fixed-activation networks, while empirical testing on language models up to 2B parameters demonstrates consistent improvements in loss metrics with minimal computational overhead.
AINeutralarXiv – CS AI · May 126/10
🧠Researchers empirically validate theoretical predictions about feature repulsion in neural network grokking, discovering that while the mathematical sign structure holds consistently across activation functions, the spectral signature of this mechanism in weight updates depends critically on activation type—appearing sharply in quadratic activations but remaining invisible in ReLU networks.
AIBullisharXiv – CS AI · Mar 176/10
🧠Researchers introduce PolyGLU, a new transformer architecture that enables dynamic routing among multiple activation functions, mimicking biological neural diversity. The 597M-parameter PolychromaticLM model shows emergent specialization patterns and achieves strong performance despite training on significantly fewer tokens than comparable models.
🏢 Nvidia
AIBullisharXiv – CS AI · Mar 27/1016
🧠Researchers from arXiv demonstrate that activation function design is crucial for maintaining neural network plasticity in continual learning scenarios. They introduce two new activation functions (Smooth-Leaky and Randomized Smooth-Leaky) that help prevent models from losing their ability to adapt to new tasks over time.
$LINK
AIBullisharXiv – CS AI · Feb 276/108
🧠Researchers propose GRAU, a new reconfigurable activation unit design for neural network hardware accelerators that uses piecewise linear fitting with power-of-two slopes. The design reduces LUT consumption by over 90% compared to traditional multi-threshold activators while supporting mixed-precision quantization and nonlinear functions.
