Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

Researchers introduce TF-RefusalBench, a multilingual benchmark measuring over-alignment in large language models used for criminal law tasks in Swiss courts. The study demonstrates that safety guardrails designed to prevent harmful outputs inadvertently compromise legitimate legal work by refusing to process content describing violent crimes, and proposes abliteration as an effective mitigation technique.

The deployment of LLMs in legal systems represents a critical intersection of AI capability and institutional constraint. Swiss Federal Supreme Court's adoption of on-premises models for translation and summarization signals growing confidence in AI's practical legal applications, yet this research exposes a fundamental tension: safety mechanisms designed to prevent misuse actively obstruct legitimate professional workflows. Lawyers reviewing criminal cases encounter detailed descriptions of violent and sexual offenses as routine job requirements, but current LLM guardrails treat such content as inherently problematic, triggering refusals and disclaimers that degrade both usability and analytical quality.

The TF-RefusalBench dataset addresses a gap in LLM evaluation methodology. Existing safety benchmarks focus on preventing harmful outputs, but rarely measure collateral damage to legitimate professional applications. The finding that over-alignment manifests differently across languages and models suggests guardrails lack contextual sophistication—they cannot distinguish between a lawyer seeking to summarize a ruling versus a bad actor seeking violent content descriptions. This multilingual, domain-specific analysis moves beyond binary right-or-wrong safety framing into practical risk management.

The demonstrated effectiveness of abliteration—surgically removing refusal mechanisms while maintaining general model performance—carries significant implications for enterprise AI deployment. Organizations with specialized, legitimate use cases now have evidence-backed approaches to customize models without wholesale fine-tuning. However, this capability also raises governance questions about model customization, validation, and oversight in regulated sectors. The research suggests future LLM safety frameworks must incorporate context awareness and professional privilege recognition rather than applying monolithic refusal strategies. This positions domain-specific, on-premises deployment as increasingly viable for high-stakes applications where safety-usability tradeoffs require careful calibration.

• →Over-alignment in LLMs causes legitimate legal professionals to face refusals when processing criminal case descriptions, degrading workflow efficiency and task faithfulness.
• →TF-RefusalBench's multilingual benchmark demonstrates over-alignment is language and model-dependent, requiring context-aware rather than blanket safety approaches.
• →Abliteration (refusal direction ablation) effectively eliminates unwanted refusals with minimal impact on general model performance and task accuracy.
• →Swiss Federal Supreme Court's operational use of LLMs validates practical deployment of AI in legal systems when safety-usability tradeoffs are properly managed.
• →Enterprise deployment of specialized LLMs requires customization frameworks that balance institutional safety requirements with professional application legitimacy.