Modality Collapse as Mismatched Decoding: Information-Theoretic Limits of Multimodal LLMs

Researchers identified a fundamental limitation in multimodal LLMs where decoders trained on text cannot effectively utilize non-text information like speaker identity or visual textures, despite this information being preserved through all model layers. The study demonstrates this 'modality collapse' is due to decoder design rather than encoding failures, with experiments showing targeted training can improve specific modality accessibility.

• →Multimodal LLMs preserve non-text information through all layers but decoders trained on text cannot effectively use it.
• →Removing 64-71% of modality-specific variance actually improves decoder performance, indicating this information acts as noise.
• →The limitation is formalized as a mismatched decoder problem bounded by Generalized Mutual Information.
• →Controlled experiments across five models confirm the bottleneck is the decoder's scoring rule, not the encoder.
• →Targeted training with specific objectives can improve modality accessibility without affecting other attributes.