AINeutralarXiv – CS AI · 9h ago6/10
🧠
Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads
Researchers have identified a structural property in Multimodal Large Language Models called functional sparsity, discovering specialized attention heads (CoRe heads) that efficiently extract relevant visual information from complex contexts. This mechanistic insight demonstrates that only the top 5% of these heads are critical for multimodal reasoning, suggesting significant potential for model optimization and inference acceleration without performance loss.