ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams

Researchers introduce ReactBench, a benchmark that exposes critical limitations in multimodal large language models' ability to reason about complex topological structures in chemical reaction diagrams. Testing 17 MLLMs reveals a 30%+ performance gap between simple anchor-based tasks and sophisticated structural reasoning tasks, indicating that visual reasoning capabilities remain fundamentally constrained despite strong semantic recognition abilities.

ReactBench addresses a critical blind spot in evaluating multimodal AI systems. While MLLMs demonstrate impressive capabilities in recognizing individual visual elements and processing straightforward linear information, the benchmark reveals they struggle fundamentally with topological reasoning—understanding how interconnected elements relate spatially and functionally. This gap matters because real-world applications from drug discovery to circuit design demand precisely this type of structural comprehension.

The research builds on growing recognition that current vision-language models excel at semantic tasks but falter on spatial reasoning. Prior benchmarks emphasized what models could identify rather than how they reason about complex relationships. Chemical reaction diagrams provide an ideal testing ground because they naturally require both local precision (recognizing individual molecules) and global coherence (understanding flow patterns and cyclic dependencies). The 1,618 expert-annotated QA pairs spanning linear chains to cyclic graphs create a rigorous evaluation framework.

The 30% performance degradation between anchor-based and holistic reasoning tasks suggests the limitation stems from reasoning capacity rather than perception—a distinction confirmed through controlled ablations. This finding has implications for scientific AI applications, where systems must navigate complexity comparable to reaction networks. For developers, ReactBench establishes measurable benchmarks for improvement, while for stakeholders investing in scientific AI, it clarifies that current generation models require architectural advances before handling sophisticated domain-specific reasoning tasks reliably.

Future work likely focuses on architectural modifications and training approaches that enhance topological understanding, potentially drawing from graph neural network research and explicit structural reasoning modules.

• →MLLMs demonstrate a 30%+ performance gap between simple recognition and complex structural reasoning tasks
• →Current models fail at fundamental tasks like endpoint counting when structures involve branching or cyclic dependencies
• →ReactBench's 1,618 annotated QA pairs provide a rigorous testbed for topological reasoning in scientific diagrams
• →Performance gaps reflect reasoning limitations rather than perception deficiencies, according to controlled ablations
• →Chemical reaction diagrams reveal that real-world scientific AI applications require architectural advances beyond current capabilities