Priming: Hybrid State Space Models From Pre-trained Transformers

Researchers introduce Priming, a method that converts pre-trained Transformers into efficient Hybrid State-Space models through knowledge transfer rather than training from scratch. The technique recovers downstream performance using less than 0.5% of original pre-training tokens and enables the first large-scale comparison of SSM architectures, with Hybrid GKA 32B achieving 3.8-point reasoning improvements while delivering 2.3x faster decoding.

Priming addresses a fundamental constraint in large-language model research: the prohibitive cost of training novel architectures from scratch. By leveraging existing pre-trained Transformers as initialization points, the method democratizes exploration of Hybrid State-Space designs that balance the contextual strengths of Attention mechanisms with the memory efficiency of recurrent models. This represents a significant shift from the traditional pre-training paradigm to a knowledge-transfer approach.

The research builds on growing recognition that Transformers, while powerful, carry computational overhead through their quadratic scaling of Key-Value caches during inference. State-Space models like Mamba offer linear scaling but traditionally lack Attention's nuanced contextual modeling. Hybrid architectures promise the best of both worlds, yet their unexplored design space has remained largely inaccessible due to pre-training costs. Priming eliminates this barrier by requiring only 0.5% of source model tokens for alignment and post-training, enabling researchers to compare architectures—Gated KalmaNet, Gated DeltaNet, and Mamba-2—under controlled conditions at scale.

The practical implications extend across multiple stakeholder groups. For inference-heavy applications, the 2.3x throughput gains translate directly to cost reductions and improved user experience. Developers gain access to a curated model zoo supporting long-context reasoning and instruction following. The industry benefits from transparency into architectural trade-offs previously obscured by heterogeneous training conditions.

The released Apache 2.0-licensed code, including optimized kernels and vLLM serving integration, signals commitment to reproducibility and broader adoption, potentially influencing how future LLM optimization research prioritizes efficiency over scale.

• →Priming converts Transformer knowledge transfer into efficient Hybrid State-Space model adaptation using <0.5% of pre-training tokens
• →Controlled architectural comparison reveals Gated KalmaNet expressiveness advantage, validated through downstream long-context reasoning performance
• →Hybrid GKA 32B achieves 3.8-point reasoning gains over source model while maintaining Transformer-level quality and enabling 2.3x decode throughput
• →Open-source model zoo and optimized kernels lower barriers for Hybrid architecture research and production deployment
• →Results suggest future large-model optimization may prioritize architectural diversity over scale through efficient knowledge transfer methods