Listener-Rewarded Thinking in VLMs for Image Preferences

Researchers introduce a listener-augmented reinforcement learning framework for training vision-language models to better align with human visual preferences. By using an independent frozen model to evaluate and validate reasoning chains, the approach achieves 67.4% accuracy on ImageReward benchmarks and demonstrates significant improvements in out-of-distribution generalization.

The challenge of training reward models for generative AI systems has long plagued the field. Current approaches either memorize training data through supervised fine-tuning or fail to generalize across diverse preference distributions. This research addresses a fundamental problem: when a model's reasoning contradicts an independent evaluator assessing the same output, the model's accuracy drops significantly. The listener-augmented GRPO framework tackles this by introducing a validation layer where a frozen vision-language model re-evaluates the reasoning chain and provides calibrated confidence scores that shape the reinforcement learning reward signal.

This approach emerges from broader trends in preference modeling and alignment. As text-to-image and text-to-video systems become more sophisticated, aligning them with nuanced human preferences at scale becomes increasingly critical. Traditional annotation pipelines are expensive and prone to inconsistency. The use of a listener model essentially creates an internal consistency check, encouraging explanations that are persuasive to independent evaluators rather than merely correct on surface-level metrics.

For the AI development community, these results have practical implications. Achieving 67.4% accuracy on ImageReward while improving out-of-distribution performance by up to 6% on large-scale datasets suggests the framework scales efficiently. The reduction in reasoning contradictions indicates models trained this way may be more interpretable and trustworthy. The promise of a data-efficient path to alignment without complex annotation pipelines could accelerate development cycles for generative model companies and researchers.

Future work should examine whether this listener-based approach extends to other domains beyond image preferences, such as video generation or multimodal reasoning tasks. The open-source release of the reasoning model enables community validation and builds on this foundation.

• →Listener-augmented GRPO framework improves reward model generalization by validating reasoning chains against independent frozen models.
• →Achieves 67.4% accuracy on ImageReward benchmark with up to 6% improvement on out-of-distribution human preference datasets.
• →Reduces reasoning contradictions between primary model and independent evaluator, enhancing model interpretability.
• →Provides scalable, data-efficient alternative to complex annotation pipelines for aligning generative models with human preferences.
• →Open-source release enables broader community adoption and validation of the approach.