To Be Multimodal or Not to Be: Query-Adaptive Audio-Visual Person Retrieval via Active Modality Detection

Researchers propose a query-adaptive audio-visual person retrieval system that intelligently detects which modalities (voice or face) are actually present in broadcast video archives, avoiding noise from absent modalities. By analyzing cross-modal score consistency, the system achieves 94.2% precision on BBC Rewind's 12,000+ videos, significantly outperforming both unimodal and fixed fusion approaches.

This research addresses a fundamental challenge in multimodal AI systems: determining when to fuse multiple data sources versus relying on individual modalities. Real-world broadcast archives present asymmetric scenarios where targets may only be heard, only seen, or both—a problem absent from curated benchmarks that assume complete data. The team's insight is elegant: when both voice and face modalities successfully identify the same person, their ranking scores exhibit high consistency; this agreement deteriorates when one modality is absent or inactive, making absence detectable through statistical patterns rather than explicit ground truth labels.

The achievement of 89% modality detection accuracy enables a system that surpasses speaker-only retrieval (82.9%) and face-only retrieval (93.4%), while substantially beating fixed fusion approaches (90.0%). The 94.2% P@1 result on BBC Rewind demonstrates the framework's practical value in production environments with millions of hours of content. The system recovers 64% of the performance gap to an oracle with perfect modality knowledge, suggesting considerable room for improvement as detection mechanisms refine.

This work has implications for video understanding systems across broadcast, surveillance, and archival applications. It highlights how assumption-aware system design—recognizing that real data differs from benchmark distributions—yields measurable improvements. The cross-modal consistency principle could generalize beyond audio-visual retrieval to other multimodal tasks where modality completeness varies. For developers building production AI systems, the research underscores that adaptive fusion strategies often outperform static architectures when handling heterogeneous real-world data.

• →Query-adaptive multimodal systems using cross-modal score consistency outperform both unimodal and fixed-fusion approaches in broadcast video retrieval.
• →The proposed framework achieves 89% accuracy detecting whether voice or face modalities are actively present in target videos.
• →Real-world broadcast archives contain asymmetric data where targets may be heard only, seen only, or both—a condition absent from curated benchmarks.
• →The system attains 94.2% P@1 on BBC Rewind corpus, recovering 64% of performance gap to oracle systems with ground-truth modality labels.
• →Cross-modal consistency principles for modality detection may generalize to other multimodal AI applications beyond person retrieval.