Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions

Researchers demonstrate that Vision-Language Models (VLMs) used for optical character recognition produce fluent but visually unsupported text, relying heavily on language priors rather than actual image content. Testing on Ancient Greek critical editions reveals VLMs generate plausible errors while traditional OCR produces local noise, with token-level grounding analysis showing model-specific vulnerabilities to hallucination.

This research exposes a critical vulnerability in how modern VLMs process visual information, particularly when applied to low-resource historical documents. The study uses controlled perturbations and conditional decoding analysis to demonstrate that even when VLM outputs appear linguistically correct, they often lack grounding in the actual visual input. The finding is significant because it challenges assumptions about VLM reliability in specialized domains where accuracy directly impacts scholarship and preservation efforts.

The research builds on growing concerns about language prior reliance in multimodal AI systems. Prior work documented this phenomenon in general contexts, but this analysis extends it to historical documents and compares behavior across model architectures. The distinction between OCR-specialist models and general-purpose VLMs proves revealing: specialist models produce fluent lexical errors with minimal image conditioning, while general VLMs maintain visual correlation even when producing wrong outputs. This suggests the problem isn't uniform across the VLM ecosystem.

For developers and organizations deploying VLMs in document processing pipelines, these findings indicate that decode-time interventions cannot reliably fix grounding failures, and post-OCR language model correction only repairs text after misrecognition occurs. This has direct implications for digital humanities projects, archival work, and any application requiring high-fidelity OCR on historical materials. The research motivates a shift toward interpretability-driven evaluation metrics beyond aggregate accuracy, which could reshape how organizations select and validate VLM deployments in production environments.

• →VLMs generate plausible but visually unsupported text due to heavy reliance on language priors rather than actual image content.
• →OCR-specialist models show weaker visual grounding than general-purpose VLMs, producing fluent errors with minimal image conditioning.
• →Decode-time interventions fail to reliably restore visual grounding in VLMs during OCR tasks.
• →Traditional OCR engines produce local noise but remain more faithful to visual input under character-level perturbations.
• →Fluent output quality does not guarantee visual grounding, requiring interpretability analysis beyond accuracy metrics.