FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation

FastSLM introduces a Hierarchical Temporal Abstractor (HTA) that compresses long-form speech into just 1.67 tokens per second—a 97% reduction—while maintaining competitive performance on speech understanding benchmarks. This architecture solves a critical scaling bottleneck for multimodal AI models by preserving acoustic detail despite extreme compression, enabling efficient deployment of speech-capable language models.

The scaling of multimodal large language models to long-form audio represents a significant technical challenge distinct from image or video processing. Audio lacks the redundancy that makes visual compression straightforward, meaning aggressive token reduction risks degrading model performance. FastSLM addresses this through hierarchical abstraction—progressively distilling acoustic features across multiple temporal scales rather than applying uniform compression, which preserves phonetic nuance while dramatically reducing computational overhead.

This work builds on the broader trend of optimizing transformer architectures for real-world deployment constraints. As MLLMs become more capable at multimodal understanding, their computational demands have grown prohibitive. Token efficiency directly impacts inference latency, memory requirements, and operational costs. FastSLM's 97% token reduction while maintaining state-of-the-art performance demonstrates that architectural innovation can overcome what appeared to be fundamental scalability tradeoffs.

For developers and enterprises deploying speech-enabled AI systems, this represents a practical pathway to lower-cost, faster inference without compromising accuracy. Organizations building conversational AI, transcription services, or real-time speech analysis can reduce infrastructure requirements substantially. The availability of open-source code and checkpoints accelerates adoption across research and production environments.

The technical contribution suggests that future MLLM architectures may increasingly separate compression strategies by modality rather than applying uniform approaches. Researchers should monitor whether hierarchical abstraction patterns emerge as standard practice for non-overlapping sequential data.

• →FastSLM achieves 97% token compression for long-form speech while maintaining competitive benchmark performance.
• →Hierarchical Temporal Abstraction preserves acoustic detail by progressively distilling features across multiple temporal scales.
• →The approach reduces computational FLOPs and parameters significantly compared to state-of-the-art models.
• →Open-source release enables rapid adoption across research and production speech AI applications.
• →Demonstrates that modality-specific compression strategies outperform uniform token reduction approaches.