GraphARC: A Comprehensive Benchmark for Graph-Based Abstract Reasoning

Researchers introduce GraphARC, a new benchmark for evaluating artificial intelligence systems on abstract reasoning tasks using graph-structured data. The framework extends the popular ARC benchmark to graph domains, revealing significant limitations in current language models—particularly a gap between understanding graph properties and executing complex transformations, with performance degrading substantially on larger instances.

GraphARC addresses a fundamental gap in AI evaluation methodology by moving abstract reasoning benchmarks beyond traditional grid and text formats into graph-structured domains. This matters because graphs represent a critical data structure across scientific computing, social networks, molecular chemistry, and knowledge systems—yet existing AI benchmarks inadequately test reasoning on this topology. The research demonstrates that state-of-the-art language models possess a measurable comprehension-execution gap: they can answer questions about graph properties but fail when required to infer and apply transformation rules, suggesting their reasoning abilities remain brittle and incomplete.

The benchmark's scalability advantage over original ARC is significant. Traditional ARC relies on hand-crafted grid puzzles, limiting dataset size and diversity. GraphARC's generative approach across diverse graph families enables systematic evaluation of how models handle increasingly complex instances—revealing that performance degrades sharply as problem scale increases. This scaling barrier mirrors concerns about current AI systems' inability to handle real-world complexity.

For the AI and machine learning community, GraphARC establishes a new evaluation standard for developing graph foundation models. The framework uniquely combines node classification, link prediction, and graph generation within one benchmark, providing a comprehensive test of structural understanding. This encourages researchers to build more robust graph-reasoning capabilities rather than optimizing for narrow task performance. The work signals growing recognition that abstract reasoning remains a frontier challenge for contemporary AI systems, despite their impressive performance on narrow benchmarks.

• →GraphARC extends abstract reasoning benchmarks to graph-structured data, filling a methodological gap in AI evaluation.
• →Current language models exhibit a comprehension-execution gap—they understand graph properties but fail at complex transformation tasks.
• →Model performance degrades significantly on larger graph instances, exposing fundamental scaling limitations.
• →The benchmark's generative approach enables systematic evaluation across diverse graph families at scale, unlike hand-crafted alternatives.
• →GraphARC provides a unified framework combining node classification, link prediction, and generation—advancing graph foundation model development.