One Model for All: Multi-Objective Controllable Language Models

Researchers introduce Multi-Objective Control (MOC), a new approach that trains a single large language model to generate personalized responses based on individual user preferences across multiple objectives. The method uses multi-objective optimization principles in reinforcement learning from human feedback to create more controllable and adaptable AI systems.

• →MOC enables one LLM to produce personalized outputs across different user preferences on the Pareto front, addressing the limitation of current RLHF methods that use fixed rewards.
• →The approach integrates multi-objective optimization principles into reinforcement learning from human feedback to train preference-conditioned policy networks.
• →MOC can fine-tune a 7B-parameter model on a single A6000 GPU, demonstrating computational efficiency improvements.
• →Experiments show MOC outperforms baselines in controllability, quality/diversity of outputs, and generalization to unseen preferences.
• →The research addresses the challenge of creating personalized LLMs despite scarce per-user data and diverse multi-objective trade-offs.