Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024

Researchers introduce Deepfake-Eval-2024, a new benchmark dataset of real-world deepfakes collected from social media in 2024, revealing that state-of-the-art detection models experience dramatic performance drops of 45-50% compared to academic benchmarks. The findings underscore a critical gap between laboratory-validated deepfake detectors and their effectiveness against actual manipulated content in circulation.

The emergence of Deepfake-Eval-2024 exposes a fundamental vulnerability in current deepfake detection infrastructure. While academic models achieve high accuracy on curated datasets, their real-world performance collapses when confronted with diverse, contemporary manipulations. This gap represents a significant security risk as generative AI technology becomes increasingly accessible, enabling bad actors to create convincing fraudulent content faster than detection systems can adapt.

The dataset's composition—spanning 52 languages, 88 websites, and multiple manipulation technologies—reflects the globalized nature of deepfake creation and distribution. Previous benchmarks, typically smaller and created under controlled conditions, failed to capture this operational complexity. The 50% accuracy decline for video detection, 48% for audio, and 45% for image models demonstrates that detection algorithms have not kept pace with evolving generation techniques.

Commercial detection models and finetuned variants show promise, though they still underperform human forensic analysts. This finding suggests that organizations managing fraud and disinformation risks cannot yet rely entirely on automated solutions. Financial institutions, social media platforms, and government agencies must maintain hybrid approaches combining algorithmic detection with human review.

The open-source nature of the dataset addresses a structural problem: detection research has relied on proprietary or outdated benchmarks, preventing meaningful progress. By releasing real-world deepfakes, researchers enable rapid iteration and development of more robust detection methods. However, this also risks enabling attackers to train evasion techniques against published detection baselines, creating an ongoing technological arms race.

• →State-of-the-art deepfake detectors experience 45-50% accuracy drops on real-world 2024 deepfakes compared to academic benchmarks.
• →The dataset covers 45 hours of video, 56.5 hours of audio, and 1,975 images across 52 languages and 88 websites.
• →Commercial deepfake detection models outperform open-source alternatives but remain inferior to human forensic analysts.
• →Academic benchmarks fail to represent the diversity and sophistication of actual deepfakes circulating on social media.
• →The performance gap creates urgent demand for improved detection methods across financial services, media verification, and content moderation.