Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection

Researchers propose 'Two Birds, One Projection,' a new inference-time defense method for Large Vision-Language Models that simultaneously improves both safety and utility performance. The method addresses modality-induced bias by projecting cross-modal features onto the null space of identified bias directions, breaking the traditional safety-utility tradeoff.

• →Current jailbreak defense frameworks for LVLMs create a tradeoff between safety and general performance on visual-reasoning tasks.
• →Researchers identified a modality-induced bias direction that undermines performance on both safety and utility tasks.
• →The proposed solution projects cross-modal features onto the null space of bias directions to remove problematic components.
• →The method requires only a single forward pass, making it computationally efficient for inference-time deployment.
• →Testing shows simultaneous improvement in both safety and utility across diverse benchmarks, breaking the conventional tradeoff.