Decoding Pedestrian Crossing Intention from Egocentric Vision via Vision Language Models

Researchers developed a method using vision language models to predict pedestrian crossing intentions from egocentric video footage, achieving state-of-the-art results through fine-tuning and incorporating contextual cues like eye gaze and ego motion. The approach frames pedestrian intent prediction as a visual question answering task and demonstrates 14.5% accuracy improvement over specialized baselines, with implications for autonomous vehicle safety systems.

This research addresses a critical gap in autonomous vehicle safety by leveraging egocentric vision—the driver's first-person perspective—to predict pedestrian crossing behavior. Traditional autonomous systems rely on third-person object detection, but pedestrian intent prediction requires understanding subtle behavioral cues only visible from a human perspective. The study's systematic evaluation of vision language models reveals that while state-of-the-art VLMs show promise, they require task-specific fine-tuning to develop genuine traffic reasoning capabilities rather than surface-level pattern matching.

The research builds on broader AI trends toward multimodal understanding and parameter-efficient adaptation. Rather than developing specialized architectures from scratch, the authors demonstrate that fine-tuning existing VLM foundations with domain-specific data yields superior results. This approach reduces computational overhead and accelerates development cycles—critical factors for real-world deployment.

The incorporation of auxiliary signals—eye gaze, ego motion, and vehicle motion—proves particularly valuable, suggesting that holistic scene understanding matters more than visual recognition alone. This finding influences how autonomous systems should be designed: safety-critical predictions benefit from integrating multiple data streams rather than optimizing single modalities. For autonomous vehicle manufacturers and safety system developers, these results validate investment in multimodal sensor fusion and indicate that VLM-based approaches offer a practical path forward.

Future work should explore real-world deployment scenarios, cross-cultural variations in pedestrian behavior, and integration with existing autonomous driving stacks. The 14.5% improvement margin represents meaningful safety gains that could reduce collision risks in urban environments where pedestrian interactions dominate.

• →Vision language models achieve state-of-the-art pedestrian intent prediction through fine-tuning rather than zero-shot performance alone.
• →Incorporating eye gaze and ego motion as contextual cues yields 14.5% accuracy improvement over baseline transformer models.
• →Egocentric vision provides safety advantages over traditional third-person object detection for understanding pedestrian behavior.
• →Fine-tuned Qwen3-VL-2B model demonstrates that parameter-efficient adaptation of VLMs outperforms specialized architectures.
• →Multimodal sensor fusion combining visual, motion, and gaze data improves traffic safety predictions significantly.