Structured Multi-Criteria Evaluation of Large Language Models with Fuzzy Analytic Hierarchy Process and DualJudge

Researchers developed DualJudge, a new framework for evaluating large language models that combines structured Fuzzy Analytic Hierarchy Process (FAHP) with traditional direct scoring methods. The approach addresses inconsistent LLM evaluation by incorporating uncertainty-aware reasoning and achieved state-of-the-art performance on JudgeBench testing.

• →Traditional direct scoring of LLMs often produces inconsistent and opaque judgments, creating a bottleneck in AI development.
• →The new Fuzzy AHP extension models uncertainty using triangular fuzzy numbers and LLM-generated confidence scores.
• →Both crisp and fuzzy AHP methods consistently outperformed direct scoring across different model scales and datasets.
• →DualJudge framework adaptively combines intuitive direct scores with structured AHP outputs for superior evaluation.
• →The research establishes uncertainty-aware structured reasoning as a more reliable pathway for LLM assessment.