WASIL: In-the-Wild Arabic Spoken Interactions with LLMs

Researchers released WASIL, a dataset of 8,529 Arabic spoken interactions with LLMs including audio, transcriptions, and user feedback, to address how speech recognition errors degrade voice assistant performance. The dataset includes a 2,000-turn test set covering Modern Standard Arabic and four dialects, with annotations distinguishing between genuine unanswerability and ASR-induced failures, enabling more accurate evaluation of voice AI systems.

WASIL addresses a critical gap in voice AI development by providing the first substantial dataset of real Arabic-language interactions with LLMs and their corresponding feedback. Voice assistants built on cascaded ASR-to-LLM pipelines face a fundamental problem: speech recognition errors propagate downstream, making it difficult to determine whether poor responses stem from model limitations or transcription failures. This dataset enables researchers to isolate these effects through gold transcripts and explicit answerability annotations.

The research reflects growing recognition that speech AI development has historically favored English and other high-resource languages, leaving Arabic speakers with comparatively limited voice assistant capabilities. Arabic presents unique challenges due to its phonetic complexity, dialectal variation, and limited training data compared to English. By releasing WASIL with samples across Modern Standard Arabic and four major dialects, the researchers provide infrastructure for more inclusive AI development.

For the broader AI industry, this work establishes a methodological template for evaluating cascaded speech systems in non-English contexts. The multi-judge LLM scoring approach offers a scalable alternative to expensive human annotation, potentially accelerating voice AI development in underrepresented language communities. The 14.2% dislike rate from real users provides baseline expectations for system performance rather than laboratory metrics.

Future development will likely see similar datasets emerge for other Arabic dialects and low-resource languages. The explicit separation of ASR-induced versus inherent errors creates clearer optimization targets for both speech recognition and language model components, advancing the technical frontier of multilingual voice interfaces.

• →WASIL dataset provides 8,529 real Arabic voice interactions with LLMs, the first major resource for evaluating cascaded ASR-LLM systems in Arabic.
• →Researchers distinguished ASR-induced errors from genuine unanswerability, enabling more precise performance diagnosis in multilingual voice systems.
• →Dataset covers Modern Standard Arabic plus four major dialects, addressing the long-tail problem of speech AI development in non-English languages.
• →Multi-judge LLM scoring approach offers scalable reference-free evaluation without expensive human annotation.
• →14.2% dislike rate from real users establishes realistic baseline expectations rather than laboratory benchmarks.