Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment

Researchers demonstrate that minimal subsets of just 50 examples (0.3% of data) can reliably evaluate large audio models with 93%+ correlation to full benchmarks. By training regression models on human-preference-aligned subsets, they achieve 98% correlation with user satisfaction—outperforming full benchmark evaluations—and release the HUMANS benchmark as an efficient LAM evaluation tool.

The research addresses a critical inefficiency in machine learning evaluation: comprehensive benchmarking of large audio models (LAMs) requires substantial computational resources and data, yet many examples contribute redundantly to performance assessment. The team's systematic analysis of 10 subset selection methods across 18 audio models and 40 tasks reveals that intelligently curated subsets dramatically reduce evaluation costs while maintaining statistical validity.

This work emerges from growing recognition that standard benchmarks don't always predict real-world user satisfaction. The researchers collected 776 human preference ratings from actual voice assistant interactions, establishing that full benchmarks achieve only 0.85 correlation with human preferences. Their innovation—training regression models on selected subsets to predict user preferences—yields 0.98 correlation, demonstrating that strategic data curation outperforms raw scale.

For AI developers and practitioners, this has immediate practical implications. Evaluating LAMs currently demands significant computational investment; showing that 0.3% of data suffices accelerates iteration cycles and democratizes model development for resource-constrained teams. The release of the HUMANS benchmark provides a standardized, efficient alternative that balances performance metrics with actual user satisfaction.

The broader significance extends beyond audio models. This methodology challenges the industry's assumption that bigger benchmarks equal better evaluation, suggesting that human-aligned, regression-weighted datasets represent the future of model assessment. As LAM development accelerates across commercial voice assistants, search, and accessibility tools, efficient evaluation frameworks become essential infrastructure.

• →Minimal subsets of 50 examples achieve 93%+ correlation with full benchmark scores, reducing evaluation data by 99.7%
• →Regression models trained on human-preference-aligned subsets outpredict full benchmarks, reaching 98% correlation with user satisfaction
• →Full LAM benchmarks show only 0.85 correlation with actual human preferences, highlighting a fundamental gap in evaluation methodologies
• →The HUMANS benchmark provides an open-source, efficient alternative for LAM evaluation that prioritizes quality-over-quantity data selection
• →This approach has broad applicability beyond audio, suggesting data curation strategy matters more than dataset scale for practical model assessment