The Rescue Effect: Spatio-Semantic Early Exit Bypasses Quantization Collapse in CLIP

Researchers address a critical failure mode in quantized Vision-Language Models by proposing LRA-EE, a technique that uses early exit strategies to bypass noise-saturated layers in INT8 CLIP. The method improves zero-shot classification accuracy by 2.44 percentage points while reducing computational load by 13.4%, demonstrating that selective layer utilization can recover performance lost to quantization-induced representation collapse.

Quantizing large neural networks for deployment on edge devices remains a fundamental challenge in machine learning infrastructure. This research identifies a previously undercharacterized problem: while traditional CNN quantization degrades classification confidence uniformly, joint-embedding architectures like CLIP suffer from directional drift in the multimodal embedding space. As noise accumulates across transformer blocks, the cosine similarity alignments that enable zero-shot retrieval deteriorate—a phenomenon the authors term Quantization-Induced Representation Collapse.

The proposed LRA-EE solution leverages an intuitive insight: not all layers contribute equally to final embeddings, and some shallow layers may encode sufficient semantic information before noise dominates. By implementing learned gating mechanisms that assess layer-specific confidence, prediction margins, and spatial activation variance, the system selectively exits early for samples that have already stabilized their representations. The four-quadrant decomposition revealing the "Rescue Effect" proves especially valuable: nearly 10% of samples actually achieve correct classification at shallow depths but lose accuracy through deeper layers—a direct cost of continuing computation through noise-saturated regions.

For practitioners deploying vision-language models in resource-constrained environments—robotics, mobile devices, edge inference—this work addresses a tangible bottleneck. The 13.4% FLOP reduction with simultaneous accuracy gains suggests efficiency gains weren't sacrificed for performance recovery. This contrasts with typical quantization trade-offs and could accelerate adoption of multimodal models in bandwidth-limited settings. The layer-adaptive calibration approach generalizes beyond CLIP, potentially benefiting other transformer-based architectures facing similar quantization challenges.

• →INT8 CLIP quantization causes directional drift in multimodal embeddings as noise accumulates across layers, distinct from traditional CNN quantization failures.
• →LRA-EE improves ImageNet zero-shot accuracy from 58.72% to 61.16% while reducing computational cost by 13.4% through selective early exits.
• →9.5% of samples achieve correct predictions at shallow layers but fail at full depth, revealing quantization noise as a primary performance degradation mechanism.
• →Learned gating based on confidence, prediction margins, and activation variance enables layer-adaptive early exit decisions calibrated to information-to-noise ratios.
• →The approach generalizes to resource-constrained deployment scenarios where multimodal models must operate within strict computational and memory budgets.