T-POP: Test-Time Personalization with Online Preference Feedback

Researchers introduce T-POP, a novel algorithm that personalizes large language models in real-time by learning from user preference feedback during text generation, without requiring parameter updates or extensive pre-existing user data. The method combines test-time alignment with dueling bandits to efficiently balance exploration and exploitation, addressing the cold-start problem in LLM personalization.

T-POP represents a meaningful shift in how personalization approaches can scale across diverse user bases. Traditional LLM personalization methods face significant friction: fine-tuning demands computational resources and time, while preference-based methods require substantial historical user data that new users simply don't possess. This cold-start problem has limited practical deployment of truly personalized AI assistants at scale.

The technical innovation here centers on decoupling personalization from model parameter updates. By learning a reward function at inference time rather than modifying the underlying LLM, T-POP enables rapid adaptation within individual conversations. The integration of dueling bandits—a sequential decision-making framework—allows the system to intelligently gather preference signals rather than passively consuming all feedback. This active learning approach means fewer user interactions are needed to capture preferences effectively.

For practitioners developing AI applications, T-POP's efficiency gains could reduce infrastructure costs and accelerate time-to-personalization. Users benefit from systems that adapt quickly without requiring lengthy training periods or accumulating extensive preference histories. The method's performance improvements over existing baselines suggest practical viability for production systems.

The broader implication centers on democratizing personalization. As LLMs become commoditized, the ability to offer genuinely personalized experiences becomes a competitive differentiator. T-POP's data-efficient approach lowers barriers for smaller organizations to implement personalization without proprietary datasets. Future development may focus on privacy-preserving variants and extending the approach to multimodal models, further broadening applicability across different user demographics and use cases.

• →T-POP enables real-time LLM personalization without modifying model parameters, solving the cold-start problem for new users
• →The algorithm uses dueling bandits to intelligently balance preference exploration and exploitation, reducing data requirements
• →Test-time alignment steers decoding processes based on learned reward functions capturing individual user preferences
• →Experimental results demonstrate rapid personalization with fewer user interactions compared to existing baseline methods
• →The approach has significant implications for scaling personalized AI systems with lower computational and data overhead