The Attacker in the Mirror: Breaking Self-Consistency in Safety via Anchored Bipolicy Self-Play

Researchers propose Anchored Bipolicy Self-Play, a new safety training method that addresses fundamental limitations in parameter-shared self-play red teaming by using distinct LoRA adapters for attacker and defender roles. The approach achieves 100x greater parameter efficiency and improved safety robustness across multiple language model scales without sacrificing reasoning ability.

Self-play red teaming has emerged as a promising approach to improve AI safety, where models learn to defend against adversarial attacks through role-based competition. However, this research identifies a critical flaw: when both attacker and defender roles share the same base model parameters, the system collapses into self-consistency, eliminating genuine adversarial pressure. The defender essentially learns to mimic the attacker rather than develop robust defenses, fundamentally undermining the safety benefits.

The solution introduces Anchored Bipolicy Self-Play, which maintains a frozen base model while training separate Low-Rank Adaptation (LoRA) modules for each role. This architectural change preserves computational stability while enforcing genuine role separation and adversarial dynamics. The method demonstrates remarkable efficiency gains, requiring 100x fewer parameters to fine-tune compared to traditional approaches, while delivering consistent improvements across Qwen2.5 models at 3B, 7B, and 14B scales.

For the AI safety and security community, this represents meaningful progress in making safety training scalable and cost-effective. The approach maintains model reasoning capabilities while improving robustness against adversarial prompts, addressing a common trade-off concern. Cross-play experiments validate that separately trained attackers and defenders genuinely outperform self-play models in adversarial scenarios, suggesting real-world applicability.

Looking forward, this method may influence how organizations approach safety fine-tuning for large language models, particularly smaller enterprises with limited computational budgets. The parameter efficiency enables broader adoption of rigorous safety testing and could accelerate deployment of safer models across applications.

• →Parameter-shared self-play causes safety dynamics to collapse into self-consistency, eliminating adversarial pressure
• →Anchored Bipolicy Self-Play achieves 100x parameter efficiency through separate LoRA adapters for attacker and defender roles
• →The method improves safety robustness across Qwen2.5 models without degrading reasoning or factual accuracy
• →Cross-play experiments demonstrate the approach produces genuinely superior adversarial defenders compared to traditional self-play
• →Lower computational costs could democratize rigorous AI safety testing for smaller organizations and research groups