Can You Break RLVER? Probing Adversarial Robustness of RL-Trained Empathetic Agents

Researchers introduce the Adversarial Empathy Benchmark (AEB) to test whether RL-trained empathetic language models remain robust against adversarial user tactics like gaslighting and emotional manipulation. While RLVER-trained models significantly outperform baselines in empathetic responsiveness, a new metric (ECS) reveals they excel at behavioral responsiveness without demonstrating genuine emotional state tracking, raising questions about the depth of empathetic AI capabilities.

This research exposes a critical gap in how empathetic AI systems are evaluated and deployed. Current benchmarks for empathetic language models assume cooperative users, but real-world interactions involve adversarial dynamics where users test, manipulate, and pressure AI systems. The study's construction of psychologically grounded adversarial scenarios—including gaslighting and emotional escalation—represents a methodologically rigorous approach to stress-testing AI systems intended for emotionally sensitive applications.

The findings present a nuanced picture. RLVER-PPO-Think models substantially outperform untuned baselines on the Adversarial Empathy Benchmark (0.963 vs. 0.761), demonstrating measurable improvements in handling hostile interactions. However, the Emotional Consistency Score (ECS) metric reveals a concerning behavioral-legibility dissociation: models improve at appearing responsive without improving at tracking underlying emotional states. This distinction matters profoundly for applications where AI systems provide emotional support or mental health assistance.

For the AI development community, these results highlight the danger of optimizing for surface-level performance metrics. RL training from emotion rewards appears to create sophisticated behavioral mimicry rather than genuine state understanding. This has immediate implications for deployment considerations—AI systems fine-tuned for empathy may present inflated confidence in their actual emotional intelligence, potentially misleading users or healthcare providers about their capabilities.

Future work must reconcile the ECS-FS gap and develop metrics that distinguish between responsive behavior and authentic state tracking. As empathetic AI moves toward clinical applications, this distinction between appearing empathetic and understanding emotions becomes ethically critical.

• →RLVER-trained models significantly outperform baselines on adversarial empathy tasks, but this masks a deeper limitation in emotional state tracking.
• →Current AI benchmarks fail to capture adversarial user behaviors like gaslighting, creating false confidence in empathetic system robustness.
• →The study's Emotional Consistency Score metric reveals empathetic AI excels at behavioral responsiveness without genuine emotional understanding.
• →RL training from emotion rewards may optimize for superficial empathy signals rather than meaningful state comprehension.
• →Clinical deployment of empathetic AI requires metrics that distinguish behavioral mimicry from authentic emotional intelligence.