Can VLMs Unlock Semantic Anomaly Detection? A Framework for Structured Reasoning

Researchers introduce SAVANT, a model-agnostic framework that improves Vision Language Models' ability to detect semantic anomalies in autonomous driving scenarios by 18.5% through structured reasoning instead of ad hoc prompting. The team used this approach to label 10,000 real-world images and fine-tuned an open-source 7B model achieving 90.8% recall, demonstrating practical deployment feasibility without proprietary model dependency.

The autonomous driving industry faces a critical challenge: detecting rare, out-of-distribution semantic anomalies that existing perception systems fail to recognize. This vulnerability poses safety risks that traditional deep learning approaches struggle to address due to the long-tail nature of edge cases. SAVANT addresses this gap by reformulating anomaly detection from black-box prompting into a principled, layered semantic consistency verification process that works across multiple VLM architectures.

The research builds on the growing recognition that VLMs possess latent reasoning capabilities underutilized by simple prompting strategies. Previous work relied heavily on proprietary models like GPT-4V, creating reproducibility issues and deployment barriers. SAVANT's two-phase pipeline—structured scene description extraction followed by multi-modal evaluation across four semantic domains—transforms anomaly detection from art into engineered methodology. The 18.5% absolute recall improvement over baseline prompting demonstrates tangible gains from structured reasoning.

The framework's real impact emerges through its data curation capability. By automatically annotating 10,000 high-confidence real-world driving images, the researchers created training data addressing the chronic scarcity problem in semantic anomaly detection. Fine-tuning Qwen2.5-VL on this dataset achieved 90.8% recall and 93.8% accuracy—surpassing all evaluated models while enabling cost-effective local deployment. This decoupling from proprietary models has profound implications for autonomous vehicle developers facing reliability and regulatory requirements.

For the AV industry, SAVANT represents a shift toward reproducible, deployable anomaly detection without vendor lock-in. The approach's model-agnostic design enables standardization across different VLM architectures. Future work likely focuses on expanding semantic domains, improving few-shot adaptation, and hardening detection against adversarial scenarios.

• →SAVANT improves VLM anomaly detection by 18.5% through structured semantic reasoning rather than ad hoc prompting.
• →The framework enabled automatic annotation of 10,000 real-world driving images with high confidence scores.
• →Fine-tuned open-source Qwen2.5-VL model achieves 90.8% recall and 93.8% accuracy, surpassing proprietary alternatives.
• →Model-agnostic design eliminates dependency on proprietary VLMs, enabling local deployment at minimal cost.
• →Structured decomposition across four semantic domains transforms anomaly detection from heuristic prompting into principled methodology.