X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs

Researchers propose X-OPD, a Cross-Modal On-Policy Distillation framework to improve Speech Large Language Models by aligning them with text-based counterparts. The method uses token-level feedback from teacher models to bridge performance gaps in end-to-end speech systems while preserving inherent capabilities.

• →End-to-end Speech LLMs show significant performance degradation compared to text-based models despite improved latency.
• →Traditional Supervised Fine-Tuning and Reinforcement Learning methods fail to close the performance gap between speech and text models.
• →X-OPD framework enables Speech LLMs to explore their own distribution through on-policy rollouts with teacher model guidance.
• →The method provides token-level feedback to distill text-based teacher capabilities into multi-modal student representations.
• →Extensive experiments show X-OPD significantly narrows performance gaps in complex tasks while maintaining model capabilities.