From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models

Researchers introduce Demo2Reward, a test-time optimization technique that improves Vision-Language Model (VLM) reward models by refining prompts based on a small number of expert demonstrations. The method reduces false positives in reward prediction without requiring additional model training, enabling more effective reinforcement learning in robotics applications including real-world scenarios.

Demo2Reward addresses a critical bottleneck in autonomous systems: the difficulty of obtaining accurate reward signals for policy learning. Traditional approaches require hand-crafted reward functions or extensive labeled data, both impractical in real-world robotics where expert demonstrations are scarce. This work leverages pre-trained VLMs as zero-shot reward models, then refines their language prompts using only 3-10 demonstration trajectories, significantly improving performance without computational overhead during policy training.

The research builds on growing recognition that foundation models contain valuable reasoning capabilities applicable beyond their original training objectives. By adapting prompts at test time rather than retraining models, Demo2Reward offers practical efficiency gains. The key innovation lies in systematically reducing false positives—incorrect reward predictions that corrupt downstream learning—while maintaining true positive detection, a nuanced optimization problem that prior zero- and few-shot approaches struggled to solve.

For robotics and AI development, this approach democratizes reward model creation by eliminating manual engineering requirements and reducing data dependency. The demonstrated transfer to real-world robotic tasks validates practical applicability beyond simulation environments. This has implications for accelerating autonomous system deployment where obtaining ground truth rewards remains expensive and time-consuming.

The work's significance extends beyond robotics to any domain requiring reward signals with limited labeled data. As foundation models become more capable, test-time adaptation methods like this unlock new capabilities without model retraining. Future developments might explore extending this technique to other VLM applications requiring task-specific optimization with minimal data.

• →Demo2Reward optimizes VLM reward models using only 3-10 demonstration trajectories without retraining or additional computation
• →The method reduces false positive predictions while preserving true positives, directly improving downstream policy learning quality
• →Real-world robotic experiments confirm the technique transfers beyond simulation, enabling learning without manual reward engineering
• →Test-time prompt adaptation emerges as practical alternative to traditional reward function design in data-scarce robotics scenarios
• →Foundation models demonstrate flexibility in adaptation for domain-specific tasks through efficient language instruction optimization