Evaluation of Small Language Models for Arabic Language Processing

Researchers evaluated 12 small language models on Arabic NLP tasks using a 240-item benchmark across 8 domains, finding that Gemma 3 (12B) performed best despite model size alone not determining performance. The study reveals that Arabic alignment and instruction-following capability matter more than parameter count, with lower-performing models struggling with prompt leakage, hallucination, and language drift.

This research addresses a critical gap in AI development: the scarcity of rigorous benchmarks for non-English languages, particularly Arabic. While large language models dominate headlines, the practical deployment of efficient, smaller models in Arabic-speaking regions requires specialized evaluation frameworks. The benchmark methodology is robust, employing multiple judge models to eliminate single-point-of-failure assessment bias and testing both comprehension and generation capabilities across diverse linguistic tasks.

The finding that model size doesn't directly correlate with Arabic performance challenges conventional scaling assumptions. This suggests that architectural choices and training data composition—specifically Arabic linguistic alignment—matter more than parameter count. Gemma 3's leadership indicates that Google's focus on multilingual pretraining produces tangible benefits, while the competitive performance of open-source alternatives like Aya demonstrates that accessibility and cultural adaptation can compete with proprietary approaches.

For the AI industry, this work validates the growing necessity of language-specific benchmarking. As deployment shifts toward edge devices and resource-constrained environments, efficient SLMs become increasingly valuable. Organizations building Arabic AI systems now have quantifiable performance baselines rather than relying on English-language metrics as proxies. The identified failure patterns—prompt leakage and language drift—are particularly relevant for production systems handling sensitive content in financial or healthcare sectors.

Looking forward, the benchmark's methodology could catalyze similar evaluations for other underrepresented languages, potentially reshaping how model developers approach multilingual training and evaluation infrastructure.

• →Gemma 3 (12B) achieved the highest performance score (4.548/5) on Arabic language tasks despite smaller size competitors.
• →Arabic language alignment and instruction-following capability matter significantly more than model parameter count alone.
• →Common failure modes include prompt leakage, hallucination, and language drift, indicating specific training gaps in Arabic models.
• →The multi-model judge framework using GPT-4 Mini, Claude Haiku, and DeepSeek-Chat provides robust evaluation methodology.
• →The benchmark establishes standardized evaluation criteria for Arabic SLMs, enabling future comparative research and development.