MuCRASP: Multimodal Chain-of-thought Reasoning aware Structured Pruning

Researchers introduce MuCRASP, a structured pruning framework designed to compress vision-language models while preserving chain-of-thought reasoning capabilities. The method addresses limitations in existing pruning techniques by identifying reasoning-critical components and accounting for differences between visual and textual modalities, achieving superior performance preservation at 30-50% compression rates.

The development of MuCRASP represents a meaningful advancement in making vision-language models more practical for deployment. Current VLMs exhibit impressive reasoning capabilities but suffer from prohibitive parameter counts that make real-world deployment economically unfeasible. Traditional pruning methods, optimized for unimodal language models, fail to account for the unique architectural demands of multimodal systems where visual and textual information processing creates distinct activation patterns.

The research identifies a critical insight: chain-of-thought reasoning depends on sparse pivot tokens that act as critical decision points in the generation trajectory. Existing pruning approaches treat all parameters equally, inadvertently damaging these reasoning pathways. MuCRASP's innovation lies in targeting reasoning-critical components while maintaining cross-modal alignment, effectively preserving the reasoning quality that makes these models valuable for complex tasks.

The empirical results demonstrate substantial practical impact. At 30% compression on Qwen2.5-VL-7B, MuCRASP achieves an LLM-as-a-Judge score of 8.87 compared to 7.32 for existing baselines on physical reasoning tasks—a meaningful gap for production systems. The framework maintains reasoning consistency even at 50% compression with lower perplexity degradation than alternatives, suggesting VLMs can be compressed more aggressively without sacrificing core capabilities.

This work matters for the broader AI infrastructure ecosystem as organizations seek to deploy advanced reasoning models cost-effectively. As VLMs become integral to enterprise applications, the ability to compress them without gutting reasoning quality directly affects inference costs, latency, and accessibility. Future development should focus on whether these pruning strategies generalize across different model architectures and whether further compression thresholds exist.

• →MuCRASP preserves chain-of-thought reasoning during model compression by identifying and protecting pivot tokens critical to reasoning trajectories.
• →The framework accounts for activation-distribution differences between visual and textual modalities, addressing a key limitation of unimodal pruning techniques.
• →At 30% pruning, MuCRASP achieves 21% higher reasoning scores than baseline methods on physical reasoning benchmarks.
• →Reasoning consistency is maintained up to 50% compression, significantly extending the practical compression range for vision-language models.
• →The method enables more cost-effective deployment of VLMs by reducing parameter counts while maintaining reasoning quality essential for complex tasks.