BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

Researchers introduce BenGER, a comprehensive benchmark dataset for evaluating large language models on German legal reasoning tasks, comprising 596 exam-style cases and 531 doctrinal reasoning problems. The study demonstrates that LLM-as-a-Judge frameworks can achieve near-human consistency in legal assessment, with human-AI collaboration substantially outperforming unaided human performance.

BenGER addresses a critical gap in LLM evaluation by creating the first large-scale benchmark specifically designed for subsumption-based legal reasoning in German law. This matters because legal reasoning demands precise argumentation and domain-specific knowledge that general-purpose LLM benchmarks fail to capture. The research validates whether contemporary language models can handle complex legal tasks that require understanding statutory interpretation, case precedent application, and structured legal argumentation.

The benchmark emerges from growing recognition that LLMs are being deployed in legal practice without adequate domain-specific evaluation. German law presents particular challenges due to its civil law tradition and language specificity, making this dataset valuable for both researchers and practitioners. The three-component structure—exam-style cases, doctrinal reasoning, and human baseline comparisons—creates a rigorous evaluation framework absent from most existing LLM assessments.

The study's most significant finding involves the LLM-as-a-Judge methodology, which achieves Calderon correlation of r=0.96 against human reviewers, matching the reliability loss from removing a single human reviewer. This suggests LLMs can supplement human legal assessment with minimal quality degradation. Notably, human-AI co-creation substantially outperforms unaided human work, indicating practical value for legal professionals using these tools as assistive technology rather than replacement systems.

The leaderboard results confirm closed-flagship systems' superiority across all evaluation metrics, suggesting that model scale and training quality remain primary determinants of legal reasoning performance. Future work should explore whether these findings generalize to other legal systems and languages, and whether performance gains in German law transfer to cross-jurisdictional reasoning tasks.

• →BenGER benchmark achieves LLM-as-a-Judge reliability comparable to human reviewers (r=0.96), validating AI use in legal assessment workflows.
• →Human-AI co-creation substantially outperforms unaided human legal reasoning, indicating significant practical applications in legal practice.
• →Closed-flagship LLM systems lead across all evaluation metrics, with open-weight models showing meaningful but secondary performance.
• →The benchmark provides the first comprehensive German law evaluation framework, enabling rigorous assessment of LLM legal reasoning capabilities.
• →Results demonstrate LLMs function better as assistive tools than independent replacements in legal contexts requiring domain expertise.