ArtiFact: A Large-Scale Multi-Modal Cultural Heritage Dataset

Researchers introduce ArtiFact, a large-scale multi-modal dataset containing 651,045 museum records from three major art institutions combined with images, text, and structured data. The dataset benchmarks AI systems on cross-modal error detection and semantic query processing tasks, revealing significant challenges in detecting domain-specific errors and handling culturally-nuanced information retrieval.

ArtiFact addresses a critical gap in AI research by providing the first large-scale, real-world multi-modal dataset specifically designed for cultural heritage applications. The dataset combines structured records, descriptive text, and images from the Metropolitan Museum of Art, the Art Institute of Chicago, and the Rijksmuseum, creating a comprehensive resource for developing robust data management systems.

The research demonstrates that current AI systems struggle with nuanced cultural and historical contexts. By injecting seven categories of errors into over 130,000 records, researchers revealed that detecting subtle domain-specific problems—such as material anachronisms where objects are described with materials not available during their purported creation period, or temporal shifts in dating—remains an unsolved challenge for machine learning models trained primarily on general-domain data.

This work has significant implications for AI development beyond cultural heritage. The findings indicate that multi-modal AI systems require domain-specific training data and approaches to handle specialized terminology, ambiguous object classifications, and historically contingent language. Museums, galleries, and cultural institutions managing digitization projects could leverage ArtiFact-trained models to improve data quality at scale, while researchers gain a benchmark for advancing multi-modal reasoning capabilities.

The dataset's release positions it as a standard evaluation tool for the database and AI communities. Future work will likely focus on developing specialized models for cultural domain tasks and exploring how cross-modal reasoning can better capture contextual information embedded in museum collections.

• →ArtiFact provides 651,045 real-world museum records as a benchmark for multi-modal AI system development
• →Current AI systems fail to detect subtle domain-specific errors like material anachronisms and temporal inconsistencies
• →The dataset reveals challenges with culturally-nuanced semantic query processing involving ambiguous terminology and historical context
• →Domain-specific AI training data is essential for handling specialized cultural heritage applications
• →This benchmark enables evaluation of cross-modal error detection and semantic understanding capabilities