Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing

Researchers introduce Dr. DocBench, a new benchmark dataset for evaluating document parsing systems on expert-level and difficult content. The dataset contains 4,514 annotated pages spanning 52 subject domains with specialized structures like chemical formulas and complex tables, revealing that state-of-the-art systems struggle significantly with these challenging real-world scenarios.

Dr. DocBench addresses a critical gap in how document parsing systems are evaluated. While existing OCR and document parsing benchmarks typically measure performance on commonly encountered documents where modern systems already excel, this new benchmark deliberately selects difficult cases where multiple state-of-the-art parsers fail. This methodology shift reflects a maturation in AI evaluation practices—moving from testing on "easy" problems to stress-testing systems on edge cases and specialized domains.

The research stems from recognition that vision-language models and document processing systems handle routine documents well, but performance degrades sharply with expert-domain content. Chemical formulas, musical notation, complex multi-page tables, and hierarchical structures present parsing challenges that generic systems struggle to solve. By curating 4,514 pages from a multilingual book corpus across 52 BISAC subject domains, Dr. DocBench creates a realistic evaluation environment reflecting actual document diversity.

For developers building document intelligence systems, this benchmark serves as a diagnostic tool revealing systematic weaknesses across content types and structural attributes. Organizations relying on document automation will find the benchmark useful for identifying gaps in their parsing pipelines before deployment. The research demonstrates that strong performance on existing benchmarks provides false confidence—systems excelling on standard tests may fail on specialized documents in production.

Looking forward, Dr. DocBench will likely drive development of more robust parsing systems that handle domain-specific content better. The benchmark's multilingual and multi-domain nature suggests future improvements must address specialized visual structures rather than optimizing for common cases. This work exemplifies how challenging benchmarks advance AI capabilities by forcing systems to handle complexity beyond typical usage patterns.

• →Dr. DocBench introduces a difficulty-aware benchmark targeting cases where state-of-the-art document parsing systems fail rather than succeed.
• →The dataset contains 65k high-quality annotations across 4,514 pages covering 52 subject domains including specialized content like chemical formulas and music notation.
• →Strong performance on existing benchmarks does not transfer to expert-level document parsing, revealing substantial gaps in current systems.
• →The multilingual benchmark spans long documents averaging 100 pages with annotations for layout, reading order, and domain-specific structures.
• →Results show pipeline-based parsers and general-purpose VLMs struggle with hierarchical relations and complex structural attributes in specialized domains.