LCSHBench: A Multilingual, Consensus-Grounded Benchmark for Library of Congress Subject Heading Assignment

LCSHBench introduces the first large-scale public benchmark for Library of Congress Subject Heading assignment, comprising 22,346 multilingual books with consensus-validated labels from three major university libraries. The dataset reveals that while libraries agree on conceptual topics 93% of the time, they differ in exact heading assignments 39.4% of the time, enabling more nuanced evaluation of automated cataloging systems.

LCSHBench addresses a significant gap in machine learning research by providing the first standardized benchmark for subject heading assignment, a task central to library science and information retrieval. The dataset's design prioritizes quality through consensus validation—records only enter when at least two independent cataloging agencies assigned identical LCSH headings, reducing noise and ensuring reliability. This methodological choice distinguishes it from typical crowdsourced benchmarks prone to annotation disagreement.

The concordance study revealing 93.3% concept-level agreement but only 60.6% exact-match agreement reflects the nuanced nature of controlled vocabulary assignment. Catalogers often choose different but semantically equivalent headings based on cataloging standards and local conventions. This insight justifies LCSHBench's dual evaluation framework scoring both exact and concept-level matches, enabling systems to be evaluated on true semantic understanding rather than surface-level string matching.

The benchmark's multilingual scope across 15 languages addresses a critical gap in non-English NLP research. The early demonstration using a 300M-parameter embedder that outperforms a larger 3,072-dimensional hosted system on exact recall suggests that efficient, locally-deployable models can match or exceed performance of larger hosted solutions. However, the authors acknowledge non-uniform cross-lingual gains, indicating substantial room for improvement.

For the AI research community, LCSHBench enables rigorous development of multilingual information retrieval and generation systems. The open-license sourcing from Harvard, Columbia, and Princeton catalogs ensures broad accessibility. Future work addressing held-out-test validation and end-to-end system evaluation will determine whether current approaches adequately solve real-world cataloging automation.

• →LCSHBench provides the first large-scale consensus-validated benchmark for automated library subject heading assignment across 22,346 books in 15 languages
• →Libraries show 93% concept-level agreement but only 60.6% exact heading match rate, indicating the need for nuanced semantic evaluation metrics
• →A fine-tuned 300M-parameter on-device embedder outperformed larger hosted embedders, suggesting efficient local models can match or exceed centralized solutions
• →The dataset's dual evaluation approach for exact and concept-level matches addresses the semantic complexity of controlled vocabulary assignment
• →Significant cross-lingual performance variation highlights remaining challenges in multilingual information retrieval systems