Direction for Detection: A Survey of Automated Vulnerability Detection and all of its Pain Points

A comprehensive survey of 87 machine learning vulnerability detection studies reveals that the field has stalled despite a decade of research, trapped in self-reinforcing feedback loops that optimize for narrow, artificial problems. Researchers identify twelve interconnected pain points spanning datasets, formulations, metrics, and evaluation approaches that perpetuate focus on binary C/C++ function-level classification while neglecting vulnerability type prediction, multilingual support, and broader detection granularities.

The automated vulnerability detection field faces a critical inflection point where decades of machine learning research have failed to produce measurable progress on real-world problems. This meta-analysis systematizes 87 influential works and exposes how the community inadvertently optimizes for proxy metrics rather than practical security outcomes. The field's concentration on binary classification of C/C++ vulnerabilities at the function level reflects not scientific necessity but rather historical accident—early dataset limitations and evaluation metrics have become entrenched, creating feedback loops where researchers train models on established benchmarks that reinforce narrow problem formulations.

The twelve identified pain points—spanning dataset construction, metric selection, baseline comparisons, and granularity mismatches—are not isolated issues but causally interconnected. Flawed datasets perpetuate inappropriate metrics; inappropriate metrics justify narrow formulations; narrow formulations limit dataset diversity. This self-reinforcing cycle explains why performance improvements on standard benchmarks have plateaued despite methodological advances in machine learning architecture and training techniques.

For software security practitioners and organizations deploying AI-driven vulnerability detection, this analysis carries immediate implications. Current ML4AVD systems optimized on academic benchmarks may not generalize to real codebases or effectively identify diverse vulnerability types beyond their training distribution. The rise of agentic coding frameworks—which accelerate code production rates—makes this limitation increasingly critical, as automated detection must now handle scale that manual verification cannot support.

Moving forward, the field requires deliberate decoupling of problem formulations from historical datasets, investment in multilingual and multitype vulnerability corpora, and alignment of evaluation metrics with real-world security outcomes rather than academic benchmarks.

• →Machine learning vulnerability detection research shows no clear performance improvement despite decade of work due to self-reinforcing feedback loops between datasets, metrics, and problem formulations.
• →The field has become trapped optimizing binary C/C++ function-level classification, neglecting vulnerability type prediction and broader language and granularity support.
• →Twelve interconnected pain points span the entire ML4AVD pipeline and perpetuate concentration on narrow, artificial problems disconnected from real-world security needs.
• →As agentic AI accelerates code production, current ML4AVD systems may fail to scale or generalize beyond academic benchmarks to production codebases.
• →Breaking feedback loops requires explicit decoupling of problem formulations from historical datasets and alignment of metrics with practical security outcomes rather than benchmark optimization.