CLIP-Inspector: Model-Level Backdoor Detection for Prompt-Tuned CLIP via OOD Trigger Inversion

Researchers introduce CLIP-Inspector, a backdoor detection method for prompt-tuned CLIP models that reconstructs hidden triggers using out-of-distribution images to identify if a model has been maliciously compromised. The technique achieves 94% detection accuracy and enables post-hoc model repair, addressing critical security vulnerabilities in outsourced machine learning services.

The outsourcing of model training to MLaaS providers creates a fundamental trust problem in AI deployment. When organizations lack resources to train vision-language models from scratch, they rely on third parties to adapt models like CLIP through prompt tuning—a process vulnerable to sophisticated backdoor attacks that existing security measures fail to detect. Unlike traditional backdoors that corrupt model encoders, these attacks leave models functionally intact while secretly redirecting specific inputs to attacker-chosen classifications, making detection extraordinarily difficult.

CLIP-Inspector addresses this critical gap by introducing a model-level verification approach that doesn't require labeled data or knowledge of attack patterns. By reconstructing potential triggers through out-of-distribution image analysis, the method achieves unprecedented detection accuracy (94% across 50 models) while operating with minimal computational overhead—processing triggers in a single epoch using just 1,000 unlabeled images. The approach demonstrates superior performance compared to adapted baselines, with AUROC scores of 0.973 versus 0.495-0.687, fundamentally changing how organizations can vet third-party models.

The ability to both detect and repair backdoored models through fine-tuning on correctly labeled triggered inputs transforms the practical security landscape for AI deployment. Organizations can now implement verification protocols before production deployment without sacrificing model quality. This creates immediate implications for enterprises relying on outsourced model adaptation, establishing a new standard for supply-chain security in machine learning that previously lacked effective detection mechanisms.

• →CLIP-Inspector detects backdoored prompt-tuned CLIP models with 94% accuracy using only unlabeled OOD images
• →The method reconstructs hidden triggers in a single epoch, making it practical for real-world deployment scenarios
• →Detected backdoors can be repaired through fine-tuning, enabling both verification and model remediation
• →Existing trigger-inversion baselines achieve significantly lower performance (0.495-0.687 AUROC vs 0.973)
• →The technique addresses supply-chain security risks in MLaaS by enabling white-box model vetting before deployment