ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models

Researchers introduced ROSE, a benchmark that evaluates how well multimodal language models can convert visual information into context-specific actions. Testing nine MLLMs revealed significant performance drops of up to 44.5 percentage points when shifting from counting tasks to region-conditioned actions, despite near-perfect human performance, indicating a fundamental gap in how these models translate perception into actionable outputs.

The ROSE benchmark addresses a critical limitation in current multimodal large language models: the ability to dynamically interpret visual scenes based on changing task contexts. While MLLMs demonstrate impressive performance on individual visual tasks, this research exposes a substantial gap between perception and action execution. The benchmark holds visual scenes constant while varying constraints and required outputs, isolating the specific failure modes in context-dependent reasoning.

This work builds on growing recognition that visual understanding alone doesn't ensure reliable task execution. Previous research has focused on individual capabilities—counting accuracy, object detection, or coordinate grounding—but ROSE examines the integration layer where models must reconcile fixed visual evidence with shifting contextual demands. The 44.5 percentage-point performance differential reveals that coordinate grounding explains only part of the problem, pointing to a distinct bottleneck in how models switch between task frames.

The implications extend across AI development priorities. For practitioners deploying MLLMs in dynamic environments—robotics, autonomous systems, or adaptive interfaces—this benchmark highlights reliability risks that benchmark suites typically miss. The research suggests current model architectures struggle with implicit reference frames and flexible action mapping, two capabilities essential for real-world deployment. Testing across nine recent models indicates this isn't a single-model limitation but a structural challenge in how these systems process context.

Future work should examine whether scaling, architectural changes, or training approaches can bridge this perception-action gap. The benchmark itself provides a controlled framework for measuring progress, enabling more precise evaluation of improvements in context-dependent visual reasoning.

• →Performance drops 44.5 percentage points when MLLMs transition from counting to region-conditioned action tasks on identical visual scenes.
• →The perception-action gap persists even when models correctly count the same objects, indicating a distinct bottleneck beyond coordinate grounding.
• →Nine tested multimodal models show consistent structural limitations in converting shared visual evidence into context-specific actions.
• →Human performance reaches 98.8% on the same tasks, establishing a clear benchmark for the reliability gap in current MLLMs.
• →ROSE provides a controlled evaluation framework for measuring progress in context-dependent visual reasoning across multimodal systems.