MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval

Researchers introduce MCERF, a multimodal retrieval framework that combines vision-language models with LLM reasoning to improve question-answering from engineering documents. The system achieves a 41.1% relative accuracy improvement over baseline RAG systems by handling complex multimodal content like tables, diagrams, and dense technical text through adaptive routing and hybrid retrieval strategies.

This research addresses a critical limitation in retrieval-augmented generation systems: their inability to effectively process the multimodal nature of technical documentation. Engineering rulebooks and standards contain interconnected textual, tabular, and visual information that traditional text-only RAG systems struggle to contextualize and retrieve accurately. MCERF's modular architecture represents a meaningful advancement in how AI systems can comprehend complex domain-specific documents.

The framework builds directly on prior work (DesignQA) but introduces significant architectural improvements through ColPali-based multimodal retrieval combined with intelligent routing mechanisms. Rather than attempting to ingest entire rulebooks, the system uses targeted strategies: explicit rule lookups for straightforward queries, vision-to-text fusion for figure and table-dependent questions, and deep reasoning modes for nuanced interpretations. This multi-pathway approach mirrors how human engineers consult documentation—different query types require different cognitive approaches.

For enterprises managing technical documentation, compliance systems, and engineering knowledge management, this represents meaningful progress toward automating complex document comprehension tasks. The 41.1% accuracy improvement is substantial for mission-critical applications where errors in technical interpretation carry real consequences. The modular design enables adoption across different model architectures, reducing vendor lock-in concerns.

The distinction between single-case and multi-agent routing approaches offers insights into scaling challenges. As these systems handle increasingly diverse query types, the routing mechanism becomes a critical bottleneck. Future work likely focuses on improving routing efficiency and extending evaluation to additional technical domains beyond the current DesignQA benchmark.

• →MCERF achieves 41.1% relative accuracy improvement by combining multimodal retrieval with adaptive query routing
• →Modular framework design enables reusability across different model architectures and technical domains
• →Vision-language models like ColPali enable simultaneous retrieval of text and visual information from engineering documents
• →The system employs four distinct reasoning strategies dynamically matched to query complexity and type
• →Framework demonstrates scalable document comprehension without requiring full rulebook ingestion