Many Preferences, Few Policies: Towards Scalable Language Model Personalization

Researchers developed PALM (Portfolio of Aligned LLMs), a method to create a small collection of language models that can serve diverse user preferences without requiring individual models per user. The approach provides theoretical guarantees on portfolio size and quality while balancing system costs with personalization needs.

• →PALM algorithm creates small portfolios of LLMs that capture representative behaviors across heterogeneous users with different preferences.
• →The method models user preferences through multi-dimensional weight vectors across traits like safety, humor, and brevity.
• →This is the first approach to provide theoretical guarantees on both size and approximation quality of LLM portfolios for personalization.
• →The research characterizes the trade-off between system cost and personalization quality in LLM deployment.
• →Empirical results validate the theoretical guarantees and show greater output diversity compared to common baselines.