Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understanding

Researchers introduce AcuLa, a post-training framework that aligns audio encoders with medical language models to enhance clinical understanding of auscultation sounds. The method leverages LLMs to generate synthetic clinical reports from audio metadata and achieves significant performance improvements across 18 cardio-respiratory tasks, including boosting COVID-19 cough detection from 55% to 89% accuracy.

AcuLa addresses a fundamental limitation in audio-based medical diagnostics: pre-trained acoustic models recognize sound patterns but lack clinical semantic understanding necessary for effective diagnosis. The framework bridges this gap through innovative audio-language alignment, treating language models as semantic teachers that instruct audio encoders about the clinical significance of detected patterns. This represents a paradigm shift in how multimodal AI can enhance domain-specific applications.

The technical approach combines representation-level contrastive learning with self-supervised modeling, enabling the system to learn clinical semantics while preserving acoustic detail. By synthetically generating clinical reports from existing metadata using LLMs, researchers solved the data scarcity problem that typically limits medical AI development. The dramatic performance improvements—mean AUROC improvement from 0.68 to 0.79 across benchmarks and the exceptional COVID-19 detection boost—demonstrate the approach's effectiveness across diverse respiratory and cardiac conditions from multiple datasets.

For the healthcare AI sector, this work establishes a scalable template for converting general-purpose pre-trained models into specialized diagnostic tools without extensive labeled data collection. This has direct implications for developing point-of-care audio diagnostics in resource-limited settings. The methodology generalizes beyond audio to other sensory modalities in medical contexts, potentially accelerating deployment of AI diagnostic assistants in clinical workflows.

• →AcuLa achieves state-of-the-art results on 18 cardio-respiratory tasks by aligning audio models with medical language models as semantic teachers.
• →The framework improves mean AUROC from 0.68 to 0.79 on classification benchmarks and COVID-19 cough detection from 0.55 to 0.89.
• →Synthetic clinical report generation using LLMs enables large-scale training without expensive manual annotation of medical audio data.
• →The approach combines contrastive learning with self-supervised modeling to preserve both clinical semantics and fine-grained temporal acoustic cues.
• →This audio-language alignment paradigm establishes a scalable template for converting pre-trained models into clinically-aware diagnostic tools.