On Wednesdays, We Ask Questions: Optimizing "Active Listening" in Automated Legal Triage and Referral

Researchers at FETCH have developed a legal triage system using low-cost LLMs to generate follow-up questions that refine legal problem classification, but found that higher-cost models like GPT-4 are necessary for generating quality plain-language questions that elicit relevant applicant information and improve classification accuracy.

The FETCH classifier represents a practical application of LLMs in legal services, addressing a critical bottleneck in legal aid systems where intake workers must quickly match applicants with appropriate legal resources. The research identifies a fundamental tension in AI cost-optimization: while large language models excel at classification with low-cost variants, the nuanced task of generating clarifying questions—which requires understanding legal contexts and plain-language communication—demands more sophisticated models. This finding has broader implications for enterprises deploying LLM ensembles; cost savings in one component can create quality bottlenecks elsewhere.

The disconnect between LLM-as-judge and human ratings reveals an important blind spot in automated AI evaluation. Legal intake questions carry high stakes—poor questions can fail to surface critical information about domestic violence or other sensitive matters, directly affecting access to justice. The researchers' discovery that certain legal categories, particularly domestic violence, show uneven fact elicitation suggests that specialized screening protocols cannot be automated uniformly across legal domains.

For the legal tech and AI industries, this study demonstrates that AI-assisted triage requires domain-specific evaluation frameworks beyond standard classification metrics. The findings suggest that organizations building legal AI systems should invest in human-expert-guided rubrics rather than relying solely on prompt engineering or cost optimization. Going forward, the effective deployment of legal AI likely depends on hybrid models combining cost-efficient LLMs for suitable tasks with premium models for quality-critical components, alongside domain-specific human oversight in sensitive practice areas.

• →Low-cost LLMs excel at classification but struggle with generating contextually appropriate plain-language follow-up questions for legal intake.
• →LLM-as-judge ratings diverge significantly from human expert evaluations, highlighting the inadequacy of automated quality assessment in legal applications.
• →Certain legal categories like domestic violence show uneven fact elicitation, suggesting some practice areas require dedicated specialized screening panels.
• →Prompt engineering alone cannot improve question quality for legal intake; architectural changes and higher-cost models are necessary.
• →Hybrid approaches combining cost-efficient and premium models alongside human oversight appear necessary for high-stakes legal AI applications.