TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech

Researchers introduce TLDR, a patch-based autoregressive framework that compresses audio tokens to accelerate text-to-speech synthesis. The method achieves 1.8x inference speedup and reduces KV-cache memory by 75% without replacing existing model modules, addressing a key efficiency bottleneck in codec-based speech language models.

TLDR tackles a fundamental computational inefficiency in modern autoregressive text-to-speech systems. Codec-based AR models generate speech as discrete token sequences that are substantially longer than their text inputs, forcing the model backbone to perform causal computation at every token position. This architectural constraint creates dual performance penalties: increased computational latency during inference and proportional growth in KV-cache memory requirements. The proposed solution elegantly sidesteps wholesale model redesign by introducing a compression layer that groups consecutive codec tokens into patch-level representations, shifting causal modeling from the token domain to a more compact patch domain.

This research emerges within a broader trend of efficiency optimization in neural speech synthesis. As TTS systems are increasingly deployed in real-world applications—from voice assistants to dubbing platforms—the inference speed-memory tradeoff has become commercially critical. TLDR's approach using lightweight compressors and LoRA adaptation demonstrates that pretrained models can be retrofitted with efficiency gains rather than requiring complete retraining.

For developers and researchers, TLDR offers immediate practical value by reducing deployment costs without sacrificing quality. The 1.8x speedup and 75% memory reduction directly lower inference infrastructure expenses for cloud-based TTS services. The patch-based methodology could inspire similar compression strategies across other sequence-modeling domains where token-level computation creates bottlenecks, particularly in audio and video processing pipelines where output sequences frequently exceed input sequence lengths.

• →TLDR achieves 1.8x inference speedup by modeling patch-level rather than token-level speech sequences
• →Global KV-cache memory reduces by up to 75% with patch size of 4, directly lowering deployment costs
• →The framework maintains TTS quality while retrofitting existing pretrained models without replacing core modules
• →Patch-based causal modeling addresses a structural efficiency bottleneck inherent in codec-based autoregressive speech synthesis
• →The approach could generalize to other sequence modeling domains with long output-to-input ratios