HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models

Researchers introduce HiPP-Prune, a new framework for efficiently compressing vision-language models while maintaining performance and reducing hallucinations. The hierarchical approach uses preference-based pruning that considers multiple objectives including task utility, visual grounding, and compression efficiency.

• →HiPP-Prune addresses the challenge of pruning vision-language models without degrading visual grounding capabilities.
• →The framework uses a hierarchical preference-conditioned approach that allows users to specify trade-offs between different objectives.
• →A visual sensitivity signal helps prevent over-pruning of vision-critical layers that enable cross-modal fusion.
• →The method optimizes for multiple objectives including task utility, hallucination robustness, compression, and stability.
• →Experiments on LLaVA demonstrate the framework can discover diverse pruning plans with controllable performance trade-offs.