PitchBench: Measuring Pitch Hearing in Audio-Language Models

Researchers introduce PitchBench, a comprehensive evaluation suite that reveals audio-language models struggle significantly with pitch hearing—a fundamental musical perception task. The benchmark's 28 experiments expose inconsistent performance across different acoustic conditions, instrument types, and response formats, indicating current ALMs lack reliable pitch perception despite their growing real-world deployment in music applications.

The emergence of audio-language models has outpaced rigorous evaluation of their core musical capabilities. While ALMs increasingly power music tutoring systems, transcription tools, and recommendation engines, the field has relied on indirect pitch assessment through higher-level tasks and multiple-choice formats. PitchBench addresses this critical gap by systematically testing what should be elementary for any music-aware AI: consistent pitch identification. The research reveals a sobering reality—frontier models fail to maintain stable pitch perception even with controlled synthetic stimuli, let alone in real-world acoustic conditions with background noise or time variations. This finding carries serious implications for deployed systems where pitch accuracy underpins entire workflows. A music transcription model that misidentifies notes creates cascading errors downstream. A tutoring system with unreliable pitch hearing cannot effectively teach music theory or provide corrective feedback. The benchmark's systematic variation of acoustic conditions—loudness, duration, instrument type, noise levels—demonstrates that ALM pitch perception is brittle and context-dependent rather than robust. The release of PitchBench as a Python package signals the research community's recognition that standardized evaluation is essential for progress. For developers building music-focused AI products, this work suggests current models may require additional fine-tuning or architectural changes before deployment in high-stakes applications. The benchmark itself becomes a development tool and a clear roadmap for improvement, establishing pitch hearing reliability as a measurable prerequisite for trustworthy audio-language systems.

• →Current audio-language models demonstrate unreliable pitch hearing across multiple acoustic conditions and sound sources
• →PitchBench's 28 experiments reveal model performance varies sharply by instrument type, note duration, and response format
• →Pitch perception remains unstable even with controlled synthetic and instrumental stimuli, not just noisy real-world audio
• →The benchmark package enables standardized evaluation and provides data generation tools for future model development
• →Music application developers should recognize current ALMs may lack sufficient pitch accuracy for professional-grade tutoring, transcription, or production workflows