DynSess: Dynamic Session-Level Evaluation and Optimization Framework for Role-Playing Agents

Researchers introduce DynSess, a framework that evaluates and optimizes role-playing agents at the session level rather than individual turns, enabling LLMs to maintain character consistency across extended conversations. The framework includes improved evaluation metrics, optimized training methods (DSPO and GSRPO), and demonstrates performance matching larger models with fewer parameters.

DynSess addresses a fundamental limitation in how conversational AI agents are currently evaluated and trained. Traditional approaches assess dialogue quality turn-by-turn, missing the holistic coherence required for extended role-playing scenarios where character consistency and interaction quality degrade over long sessions. This research shifts the paradigm toward session-level evaluation, capturing emergent behaviors that only manifest across multi-turn interactions.

The technical approach combines three components: DynSess-Eval provides human-aligned scoring of complete dialogue sessions, multi-turn lookahead search generates high-quality training trajectories, and two complementary optimization variants (DSPO for off-policy and GSRPO for on-policy) improve agent performance. The framework's significance lies in demonstrating that session-level training produces more parameter-efficient models without sacrificing quality—a critical consideration for deploying conversational agents at scale.

For the AI industry, this work has immediate implications for conversational AI applications including customer service, virtual assistants, and interactive entertainment. The improved evaluation metrics provide developers with more reliable tools for assessing real-world performance. The parameter efficiency breakthrough suggests that future conversational agents can deliver comparable quality with reduced computational overhead, lowering deployment costs and environmental impact.

Future research directions include extending session-level optimization to other dialogue tasks, investigating scaling properties with larger models, and exploring how these techniques transfer across different character domains and interaction styles. The planned release of datasets and code will accelerate adoption of session-level evaluation standards within the research community.

• →DynSess shifts dialogue evaluation from turn-level to session-level, capturing character consistency across extended conversations.
• →The framework achieves performance parity with larger models while using substantially fewer parameters, improving efficiency.
• →Session-level training data generation through multi-turn lookahead search enables higher-quality optimization trajectories.
• →Human evaluations confirm DynSess-Eval aligns better with human judgments than existing evaluation methods.
• →Open-source release of datasets and code will facilitate broader adoption of session-level evaluation in conversational AI research.