Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York City

Researchers benchmark Vision Language Models (VLMs) and human drivers from Lima and New York City on autonomous driving comprehension tasks using dashcam footage, finding that VLMs and humans diverge in responses but geography has minimal impact due to the extreme out-of-distribution nature of challenging driving scenarios in these underserved markets.

The Robusto-2 study addresses a critical gap in autonomous vehicle development: how well VLM-based cognitive systems generalize to geographies where no self-driving companies currently operate. By conducting factorial analysis across human drivers and AI models in Lima and NYC—both notoriously challenging driving environments—the researchers reveal fundamental differences in how humans and VLMs process driving scenarios.

This research emerges as autonomous vehicle technology scales globally, with companies increasingly relying on VLMs as decision-making backbones rather than traditional rule-based systems. The choice of Lima and NYC reflects a pragmatic focus on underserved markets with complex, unpredictable traffic patterns, construction, informal street vendors, and vehicles that defy standard datasets. These cities represent real deployment challenges that simulation and sanitized datasets cannot capture.

The key finding—that humans answer consistently regardless of origin while VLMs show systematic divergence—suggests VLMs lack the cross-cultural driving intuition humans naturally develop. Surprisingly, geography itself proved less predictive than question type, indicating that extreme OOD scenarios overwhelm geographic specificity. This has profound implications for autonomous vehicle safety certification, as it suggests current VLMs may fail unpredictably in novel environments rather than exhibiting predictable regional biases.

For the industry, this work establishes benchmarking standards for emerging markets and exposes gaps in VLM reasoning under edge cases. Developers and regulators should prioritize diverse geographic datasets and stress-testing before deploying autonomous systems internationally. The public dataset release enables community-wide improvements in handling OOD scenarios.

• →VLMs and human drivers show divergent responses to driving scenarios despite comparable geographic familiarity patterns
• →Question type significantly modulates response differences more than geographic origin of respondents
• →Lima and NYC driving scenarios represent extreme out-of-distribution cases that challenge current VLM generalization
• →Human drivers from different geographies demonstrate consistent answering patterns, suggesting shared driving intuition
• →Current VLM-based autonomous systems may face unpredictable safety challenges in underserved markets lacking training data