From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning

Researchers propose a dynamic agent-centric benchmarking system for evaluating large language models that replaces static datasets with autonomous agents that generate, validate, and solve problems iteratively. The protocol uses teacher, orchestrator, and student agents to create progressively challenging text anomaly detection tasks that expose reasoning errors missed by conventional benchmarks.

• →Traditional static datasets for LLM evaluation are limited in scalability and fail to capture evolving AI reasoning capabilities.
• →The new protocol uses three agent roles: teacher generates problems, orchestrator validates them, and student attempts solutions.
• →The benchmark automatically scales in difficulty as more capable agents are introduced, enabling progressive evaluation without manual dataset curation.
• →Text anomaly detection format is used to test cross-sentence logical inference while resisting pattern-matching shortcuts.
• →The approach systematically reveals corner-case reasoning errors that standard benchmarks cannot detect.