DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following

Researchers introduce DIALEVAL, a new automated framework that uses dual LLM agents to evaluate how well AI models follow instructions. The system achieves 90.38% accuracy by breaking down instructions into verifiable components and applying type-specific evaluation criteria, showing 26.45% error reduction over existing methods.

• →DIALEVAL automates instruction evaluation using dual LLM agents and type-theoretic framework without manual annotation.
• →The system decomposes instructions into typed predicates with formal atomicity and independence constraints.
• →Framework applies differentiated evaluation criteria based on predicate types, mirroring human assessment patterns.
• →Achieves 90.38% accuracy with 26.45% error reduction compared to baseline evaluation methods.
• →Extended functionality supports multi-turn dialogue evaluation through history-aware satisfaction functions.