A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision

Researchers introduce TGAD, a new benchmark for evaluating text-guided anomaly detection systems, revealing that current multimodal vision-language models do not actually use language instructions to condition their decisions as claimed. Testing shows that removing object nouns causes performance to collapse, and component-level instructions fail to constrain defect detection, suggesting these systems rely primarily on visual features rather than genuine language guidance.

The research addresses a critical gap between marketing claims and actual capabilities in multimodal anomaly detection systems. While vendors present text-guided zero- and few-shot inspection as enabling language-controlled industrial quality assurance, existing benchmarks inherited from unimodal tasks hold text constant, making it impossible to measure whether language actually influences decisions. The TGAD benchmark progressively tests language's functional role across three scenarios: prompt sensitivity on standard datasets, component-specific instructions on extended datasets, and a realistic industrial assembly panel scenario requiring both defect and location knowledge.

The findings are sobering. Generative vision-language models showed catastrophic performance drops when object nouns were removed from prompts (97.4 to 82.6 I-AUROC), indicating language superficially conditions outputs. Component-level instructions failed to constrain decisions when defects appeared outside instructed regions, dropping from 90.3 to 66.3 accuracy. Most concerning, when combined demands appeared on the new Assembled Panel Dataset, performance collapsed below chance in some cases (31.5 versus random 50 percent baseline).

These results expose a fundamental mismatch between research narratives and deployment readiness. Industrial applications requiring reliable language-based control cannot depend on systems that absorb textual input without meaningfully altering visual-feature-based decisions. The research suggests current multimodal anomaly detection systems function primarily as vision models with language interfaces rather than genuinely integrated multimodal systems. This finding matters for industrial automation vendors and enterprises considering these tools for quality control, as reliability depends on whether language instructions can actually override visual pattern recognition.

• →Multimodal anomaly detection systems absorb language input without genuinely conditioning decisions on textual content.
• →Removing object nouns from prompts causes generative model performance to drop from 97.4% to 82.6%, revealing language dependency is superficial.
• →Component-level instructions fail to constrain defect detection when anomalies exist outside instructed regions, dropping accuracy from 90.3% to 66.3%.
• →Performance collapses below chance (31.5%) when systems face combined defect-type and component-location requirements on realistic industrial datasets.
• →Standard benchmarks overstate text-guided capabilities, requiring new evaluation protocols before these systems are safe for industrial deployment.