Online Agent-as-a-Judge: Situation-Generating Evaluation for Interactive Agents

Researchers propose Online Agent-as-a-Judge, a new evaluation framework that uses an in-world evaluator agent to actively test LLM-powered interactive agents across specific social scenarios. Unlike passive evaluation methods, this approach generates targeted situations to reveal behaviors that might otherwise remain unobserved, improving assessment reliability in complex multi-agent environments.

The development of Online Agent-as-a-Judge addresses a critical limitation in current LLM agent evaluation methodologies. Traditional approaches passively observe agent behavior in simulated environments and score the resulting trajectories, but this hands-off stance fails to systematically probe for capabilities under specific social conditions. The framework tackles this gap by deploying an evaluator agent that actively engages the target agent through dialogue and actions, deliberately creating situations designed to elicit behaviors relevant to predefined evaluation criteria.

This research emerges from growing recognition that LLM-powered social agents require more sophisticated testing than language models used for isolated tasks. Social competence depends heavily on context, relationship history, and role dynamics—factors that random or unguided interactions may never trigger. The paper demonstrates this in a life-simulation environment with 32 designer-authored social criteria, showing that active situation generation significantly improves coverage and alignment with human judgment compared to passive methods.

The implications extend across AI development and deployment. Companies building conversational agents for customer service, education, or entertainment need reliable ways to verify social competence before release. This framework provides a more rigorous evaluation methodology that reduces the risk of deploying agents with undetected behavioral gaps. For researchers, it establishes a new paradigm where evaluators become active participants rather than passive observers.

Looking forward, the challenge lies in scaling this approach beyond controlled simulation environments to real-world deployment scenarios. Future work should explore how criterion design affects evaluation validity and whether insights transfer across different types of social domains.

• →Online Agent-as-a-Judge uses active evaluator agents to deliberately generate specific social scenarios, revealing capabilities that passive methods miss
• →The framework improved evaluation criteria coverage and agreement with human labels in life-simulation testing with 32 social criteria
• →Traditional passive evaluation methods fail to systematically probe for behaviors only observable under specific social circumstances like conflict resolution
• →This methodology addresses a critical gap in LLM agent assessment for socially-dependent competencies tied to context and interaction history
• →The approach enables more reliable evidence-grounded evaluation before deploying interactive agents in customer-facing or educational applications