CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents

Researchers have created CzechTopic, a new benchmark dataset for evaluating AI models' ability to identify specific topics within historical Czech documents. The study compared various large language models and BERT-based models, finding significant performance variations with the strongest models approaching human-level accuracy in topic detection.

• →CzechTopic introduces the first human-annotated benchmark for zero-shot topic localization in Czech historical documents.
• →Large language models showed substantial performance variability, from near-human accuracy to significant failures in text span identification.
• →Smaller distilled token embedding models remained competitive despite their reduced scale compared to large language models.
• →The evaluation framework measures performance against human agreement rather than single reference annotations.
• →The dataset and evaluation tools are publicly available for research use.