Towards Realistic Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions

Researchers have introduced RealPref, a new benchmark for evaluating how well Large Language Models follow user preferences in long-term personalized interactions. The study reveals that LLM performance significantly degrades with longer contexts and more implicit preference expressions, highlighting challenges in developing user-aware AI assistants.

• →RealPref benchmark includes 100 user profiles and 1300 personalized preferences to test LLM preference-following abilities.
• →LLM performance drops significantly as conversation context length increases and preferences become more implicit.
• →The benchmark features four types of preference expression ranging from explicit to implicit communications.
• →Current LLMs struggle to generalize user preference understanding to previously unseen scenarios.
• →The research provides foundation for developing more adaptive personal AI assistants.