LithoBench: Benchmarking Large Multimodal Models for Remote-Sensing Lithology Interpretation

LithoBench introduces a comprehensive benchmark dataset for evaluating large multimodal models on remote-sensing lithology interpretation, containing 10,000 expert-annotated instances across cognitive levels from identification to reasoning. The research reveals significant gaps in current vision-language models' ability to handle knowledge-intensive geological tasks, highlighting the challenges of applying general-purpose AI to specialized domain expertise.

LithoBench addresses a critical gap in AI evaluation frameworks by creating a specialized benchmark for geological remote sensing. While large language and vision-language models have achieved impressive performance on general tasks, their application to specialized scientific domains remains underexplored. This research demonstrates that commodity AI models struggle with the nuanced, expert-level reasoning required for lithological interpretation—a task that demands understanding subtle visual cues, spectral data, textural patterns, and contextual geological knowledge simultaneously.

The benchmark's multi-level cognitive structure reflects realistic professional workflows, progressing from simple identification tasks to complex mechanism explanation and comprehensive reasoning. By organizing 10,000 instances across five cognitive tiers and using expert-in-the-loop validation, the researchers establish a rigorous evaluation standard that captures domain-specific requirements general benchmarks overlook. This approach has broader implications for scientific AI evaluation methodology.

The findings expose a significant opportunity in the AI development landscape. Organizations building specialized models for geology, mining, and resource exploration could gain competitive advantages by fine-tuning models on curated geological datasets rather than relying on general-purpose solutions. Industries dependent on geological surveys—including mineral exploration, petroleum prospecting, and infrastructure planning—may need to invest in domain-specialized model development or partnerships.

LithoBench establishes a foundation for advancing AI capabilities in geoscience. Future work should focus on whether specialized training datasets and domain-adapted architectures can close the performance gap, and whether similar benchmarking approaches could accelerate AI adoption across other scientific and technical fields where expert reasoning currently remains irreplaceable.

• →Large vision-language models demonstrate substantial limitations in geological semantic understanding, particularly for higher-order reasoning and application tasks.
• →LithoBench's 10,000 expert-annotated instances across five cognitive levels provide the first rigorous evaluation framework for lithology interpretation AI.
• →The research indicates specialized domain benchmarks are necessary to properly evaluate AI performance beyond general-purpose tasks.
• →Geological survey and mineral exploration industries may require domain-specialized model development rather than relying on commodity AI solutions.
• →Expert-in-the-loop benchmark construction enhances geological validity and establishes reproducible evaluation standards for scientific AI applications.