SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety

Researchers have developed SafeDPO, a simplified approach to training large language models that balances helpfulness and safety without requiring complex multi-stage systems. The method uses only preference data and safety indicators, achieving competitive safety-helpfulness trade-offs while eliminating the need for reward models and online sampling.

• →SafeDPO provides a lightweight alternative to complex reinforcement learning methods for AI safety alignment.
• →The approach requires only one additional hyperparameter and minimal modifications to existing training methods.
• →Testing on PKU-SafeRLHF-30K benchmark shows substantial safety improvements while maintaining helpfulness.
• →The method scales effectively to large language models with up to 13 billion parameters.
• →SafeDPO eliminates the need for reward models, cost models, and online sampling during training.