COMET: Concept Space Dissection of the Modality Gap in Audio-Text Multimodal Contrastive Embeddings

Researchers introduce COMET, a PLS-SVD framework that analyzes the modality gap in Contrastive Language-Audio Pretraining (CLAP) models by decomposing embeddings into interpretable concepts. The study reveals that only a small subset of shared conceptual axes drives similarity computation, and proposes a training-free spectral truncation method that improves zero-shot audio captioning performance while reducing dimensionality.

CLAP models have become foundational for audio understanding tasks, yet their effectiveness is fundamentally constrained by the modality gap—a persistent misalignment between audio and text embedding spaces. Prior research attributed this gap primarily to the cone effect, focusing on mean embedding shifts as the primary culprit. However, incremental improvements from mean correction alone suggested deeper structural issues remained unexplored. This research addresses that gap by introducing a novel analytical framework that reframes the problem through the lens of concept decomposition rather than simple statistical corrections.

The COMET framework uses partial least squares singular value decomposition to systematically dissect CLAP embeddings, revealing that meaningful similarity computation relies on a surprisingly small number of interpretable conceptual axes. This finding has significant implications for both researchers and practitioners. The work demonstrates that the modality gap is not monolithic but rather a multifaceted phenomenon involving information imbalance and dimensionality collapse—hypotheses that had previously lacked rigorous validation in the audio domain.

The practical impact of this research manifests through the proposed spectral truncation method, which operates without requiring model retraining, auxiliary memory banks, or substantial computational resources. This makes the approach immediately applicable to existing deployments. The method's success in enabling zero-shot audio captioning with condition swapping to approach fully supervised performance levels suggests meaningful progress toward more efficient multimodal systems. Additionally, achieving substantial dimensionality reduction while maintaining retrieval and captioning performance has implications for model deployment and resource efficiency.

Future development likely focuses on applying these insights to other modality combinations and exploring whether similar concept-based decomposition patterns emerge across different contrastive learning architectures.

• →COMET framework reveals that only a small subset of shared conceptual axes in CLAP embeddings drives meaningful similarity computation, contradicting assumptions about modality gap structure.
• →Training-free spectral truncation method achieves near-supervised performance on zero-shot audio captioning without requiring auxiliary memory or expensive computation.
• →The modality gap involves multiple factors beyond mean shift, including information imbalance and dimensionality collapse, now systematically validated in the audio domain.
• →Embedding dimensionality can be substantially reduced while preserving strong performance on both retrieval and captioning tasks through concept-based analysis.
• →Framework's applicability to existing CLAP deployments without retraining enables immediate practical improvements to audio understanding systems.