PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks

Researchers introduce PolyGLU, a new transformer architecture that enables dynamic routing among multiple activation functions, mimicking biological neural diversity. The 597M-parameter PolychromaticLM model shows emergent specialization patterns and achieves strong performance despite training on significantly fewer tokens than comparable models.

• →PolyGLU allows transformer neurons to dynamically choose between 4 activation functions, inspired by biological neurotransmitter diversity.
• →The model develops emergent depth-dependent specialization with early layers preferring GELU and deep layers favoring Tanh activation.
• →Training requires only a single A100 GPU and adds minimal 0.23% parameter overhead while maintaining robustness during fine-tuning.
• →PolychromaticLM achieves 62-89% of Qwen3-0.6B performance despite training on 3,600x fewer tokens.
• →All code, weights, and training infrastructure are released under Apache 2.0 license for open research.