PerMix-RLVR: Preserving Persona Expressivity under Verifiable-Reward Alignment

Researchers introduce PerMix-RLVR, a training method that enables large language models to maintain persona flexibility while preserving task robustness. The approach addresses a fundamental trade-off in reinforcement learning with verifiable rewards, where models become less responsive to persona prompts but gain improved performance on objective tasks.

The research tackles a critical challenge in LLM alignment: the tension between robustness and expressivity. Traditional persona prompting requires inference-time optimization to find effective character assignments, creating computational overhead. While reinforcement learning with verifiable rewards (RLVR) improves model robustness on goal-oriented tasks by reducing sensitivity to prompt variations, it inadvertently diminishes persona fidelity—the model's ability to authentically adopt requested character traits during role-playing scenarios. This creates a genuine optimization dilemma for developers seeking models that perform reliably while remaining adaptable.

PerMix-RLVR addresses this by strategically mixing persona-augmented training data during the reinforcement learning process, allowing models to maintain strong robustness against harmful persona variations while preserving faithful character adoption when appropriate. The methodology demonstrates measurable improvements: 21.2% increase in persona stability on MATH500 benchmarks and 11.4% improvement in persona fidelity on PersonaGym evaluations.

For the AI development community, this represents meaningful progress in training efficiency and model flexibility. Rather than relying on expensive inference-time prompt optimization, developers can train models that inherently handle persona variation effectively. The work also contributes to understanding outcome-based optimization trade-offs—a fundamental concern as AI systems become more autonomous and specialized.

The implications extend to diverse applications from educational AI assistants to customer service systems where consistent performance and adaptive personality are simultaneously valuable. Future research likely builds on this foundation to balance additional competing objectives in aligned LLMs.

• →PerMix-RLVR improves persona stability by 21.2% while enhancing persona fidelity by 11.4% compared to standard RLVR approaches
• →Reinforcement learning with verifiable rewards creates an inherent trade-off between robustness and persona expressivity
• →The training-time persona-mixing strategy eliminates costly inference-time prompt optimization procedures
• →The method enables models to resist harmful persona manipulations while maintaining authentic character adoption when needed
• →This research advances understanding of multi-objective optimization challenges in aligned language model training