Multi-Legal-Bench: Evaluating LLMs on Legal Reasoning Across Jurisdictions, Languages, and Legal Traditions

Researchers introduce Multi-Legal-Bench, a cross-jurisdictional benchmark evaluating large language models on legal reasoning tasks across six European countries, four language families, and 134 million court decisions. The study reveals that few-shot transfer effectiveness depends on label-set alignment rather than linguistic proximity, and that model architecture matters more than tokenizer efficiency for cross-lingual legal NLP performance.

Multi-Legal-Bench addresses a critical gap in LLM evaluation: most legal NLP benchmarks test single languages or aggregate incomparable tasks across jurisdictions, preventing meaningful cross-cultural analysis. This research establishes the first standardized framework measuring identical legal reasoning tasks—court classification, judgment forms, case outcomes, norm extraction, and cause prediction—across Ukraine, France, Netherlands, Poland, Czech Republic, and Lithuania. The sparse 5x6 matrix design deliberately avoids forcing irrelevant comparisons while enabling genuine cross-jurisdictional insights.

The findings challenge conventional assumptions about multilingual AI. Few-shot prompting effects discovered in Ukrainian legal tasks replicate across all tested jurisdictions, suggesting robust cross-cultural learning patterns. However, language family proximity fails to predict transfer quality; Ukrainian-to-French transfer (-2.1 percentage points) outperforms Ukrainian-to-Polish despite Polish being linguistically closer. This counterintuitive result indicates that label-set alignment and training data composition drive performance more than linguistic kinship.

For the AI development community, these results have significant implications for deploying legal LLMs in non-English jurisdictions. The weak correlation between tokenizer efficiency and cross-lingual accuracy (r=-0.27) indicates that practitioners cannot rely on character-level optimizations; instead, they must focus on architectural choices and pretraining data curation. The benchmark's release of 134 million court decisions, prompts, and model predictions establishes a foundation for developing jurisdiction-aware legal AI systems that could reduce access barriers to legal services across Europe.

• →Few-shot learning effects in legal reasoning tasks transfer reliably across six European jurisdictions and multiple language families.
• →Label-set alignment predicts cross-lingual transfer quality better than linguistic proximity, challenging common multilingual NLP assumptions.
• →Tokenizer efficiency has minimal predictive power for cross-lingual accuracy, suggesting model architecture and pretraining data are dominant factors.
• →No single LLM dominates across all language-task combinations, indicating jurisdiction-specific model selection remains necessary.
• →The benchmark's 134 million court decisions provide a novel resource for training jurisdiction-aware legal AI systems.