DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency

Researchers introduce DINORANKCLIP, an advanced vision-language pretraining framework that improves upon CLIP by incorporating DINOv3 distillation and high-order ranking consistency. The method addresses fundamental limitations in contrastive learning by preserving fine-grained visual details and implementing a third-order Plackett-Luce ranking model, achieving consistent improvements across benchmarks with modest computational requirements.

DINORANKCLIP represents a meaningful refinement in vision-language model architecture, targeting specific weaknesses that have persisted in CLIP-based approaches. The research identifies two critical structural issues: the symmetric InfoNCE loss discards ordering information among unmatched image-text pairs, and global pooling sacrifices sensitivity to fine-grained local structures. Previous work like RANKCLIP addressed only the first problem while remaining computationally expensive and theoretically limited to first-order interactions.

The key innovation involves injecting a frozen DINOv3 vision encoder through a lightweight dual-branch student architecture combined with multi-scale fusion and attention mechanisms. This preserves rich local structural information while maintaining cross-modal alignment. More significantly, the paper proposes a higher-order ranking model where per-position utilities include pairwise and tuple-wise transition terms, demonstrating that optimal performance occurs at third order across evaluated benchmarks. This framework elegantly contains both CLIP and RANKCLIP as special cases, providing theoretical grounding for the architecture.

The empirical validation is particularly noteworthy for its efficiency: the complete study—including order sweeps, fine-grained probing on five datasets, modality-gap analysis, and ablation studies—executes within 72 hours on a single H100 eight-GPU node using only Conceptual Captions 3M. DINORANKCLIP outperforms established baselines (CLIP, CyCLIP, ALIP, RANKCLIP) under matched computational budgets, with pronounced gains in fine-grained reasoning and out-of-distribution evaluation—precisely where local structural understanding matters most. This work suggests that vision-language models can achieve better performance through principled architectural improvements rather than scale alone, potentially influencing how future multimodal systems are designed.

• →DINORANKCLIP integrates DINOv3 distillation with high-order ranking consistency to overcome CLIP's architectural limitations in preserving fine-grained visual information.
• →The optimal ranking model order is 3 across benchmarks, demonstrating that higher-order tuple interactions improve vision-language alignment beyond first-order approaches.
• →The framework achieves superior performance compared to CLIP, RANKCLIP, and other baselines while training efficiently on standard hardware in under 72 hours.
• →Fine-grained and out-of-distribution evaluations show the largest relative improvements, indicating better local structural reasoning capabilities.
• →The modular design allows DINOv3 features to be injected without disrupting cross-modal alignment, suggesting a practical path for incorporating stronger vision encoders.