VocSim: A Training-free Benchmark for Zero-shot Content Identity in Single-source Audio

Researchers introduce VocSim, a training-free benchmark for evaluating audio embeddings' ability to identify content across diverse sound sources without parameter updates or labeled data. Testing 125k clips spanning speech, animal vocalizations, and environmental sounds, the study reveals that while frozen Whisper embeddings perform well overall, significant generalization gaps exist for low-resource and non-English languages, with implications for audio AI model development.

VocSim addresses a critical gap in audio AI evaluation by introducing a zero-shot benchmark that measures intrinsic embedding quality without fine-tuning. Unlike traditional supervised classification metrics that rely on parameter updates, this approach directly assesses whether general-purpose audio representations naturally cluster semantically similar content. The benchmark aggregates 125,000 single-source audio clips across 19 datasets, creating a comprehensive testbed that isolates content representation from source separation challenges inherent in polyphonic audio.

The research reveals important performance patterns across audio domains. Frozen Whisper features combined with time-frequency pooling and label-free PCA whitening achieve competitive zero-shot results with stable rankings across different sound categories (Kendall's tau = 0.60). However, critical weaknesses emerge in cross-lingual speech scenarios, particularly with low-resource languages like Shipibo-Conibo and Chintang, where local retrieval performance deteriorates substantially despite remaining above random chance baselines.

These findings hold significant implications for the audio AI ecosystem. The identified cross-lingual generalization gap highlights fundamental limitations in current embedding models, suggesting that commercial speech applications relying on these systems may perform poorly for endangered or underrepresented languages. The benchmark's external validation—predicting avian perceptual similarity, improving bioacoustic classification, and achieving state-of-the-art HEAR benchmark results—demonstrates practical relevance beyond academic interest.

Moving forward, developers building audio systems should prioritize testing against VocSim's public leaderboard to understand representation quality in zero-shot settings. The work signals that future improvements must specifically address language diversity and low-resource scenarios to achieve genuinely universal audio understanding.

• →VocSim introduces a training-free benchmark using 125k audio clips to evaluate zero-shot content identity without parameter updates or labeled data.
• →Frozen Whisper embeddings show strong performance overall but exhibit significant generalization failures on low-resource and non-English speech.
• →Label-free PCA whitening provides effective anisotropy correction, enabling stable performance rankings across diverse sound domains.
• →The benchmark's external validation demonstrates practical value for avian bioacoustics, bioacoustic classification, and the HEAR benchmark.
• →A cross-lingual speech generalization gap exposes critical limitations in current embedding models for endangered and underrepresented languages.