Large Language Models Should Learn Personalized Rather Than Aggregated Human Preferences
A position paper argues that large language models should optimize for individual user preferences rather than aggregated 'average user' preferences, which masks critical information about preference diversity and values. The authors propose bounded personalization frameworks that balance individual autonomy with universal safety constraints, while addressing scalability and manipulation risks.
This academic position paper challenges a fundamental assumption in current LLM alignment strategies: that averaging diverse human preferences into a single reward signal produces optimal outcomes. The authors ground their critique in social choice theory, demonstrating that aggregation systematically erases information about individual values, demographic variation, and contextual dependencies that real users find important.
The personalization debate reflects a broader tension in AI development between universalist and pluralistic design philosophies. Current approaches assume a one-size-fits-all safety and value framework works best, but this masks genuine disagreement about what constitutes harmful or beneficial outputs across different communities and contexts. The paper surveys technical solutions ranging from preference elicitation methods to modular model architectures capable of adapting to individual users without requiring separate model instances.
For the AI development community, this argument carries significant implications. Personalization could improve user satisfaction and trust by respecting legitimate value differences, while potentially reducing the political friction around whose preferences get baked into default model behavior. However, it introduces measurable safety challenges: filter bubbles that reinforce existing beliefs, value lock-in preventing users from exploring alternative perspectives, and sophisticated psychological manipulation through preference targeting.
The practical path forward involves research into bounded personalization frameworks that preserve hard safety constraints—preventing illegal content or demonstrable harms—while allowing flexibility on subjective values and communication styles. This requires better mechanisms for expressing user preferences, auditing personalized behavior at scale, and establishing clear regulatory boundaries. The success of this approach depends on whether the AI research community can engineer guardrails that prevent malicious personalization while enabling benign individual variation.
- →Aggregating human preferences into single reward signals masks critical information about individual values and preference diversity across demographic groups
- →Bounded personalization frameworks could improve user autonomy and satisfaction while maintaining universal safety constraints on harmful outputs
- →Personalized LLMs introduce genuine risks including filter bubbles, value lock-in, and sophisticated manipulation that require active mitigation strategies
- →Current alignment approaches assume one-size-fits-all values, creating political friction around whose preferences become default model behavior
- →Implementing preference-aware models at scale requires advances in preference elicitation, behavioral auditing, and regulatory frameworks to prevent malicious personalization