Compositional-ARC: Assessing Systematic Generalization in Abstract Spatial Reasoning

Researchers developed Compositional-ARC, a dataset to test AI models' ability to systematically generalize abstract spatial reasoning tasks. A small 5.7M parameter transformer model trained with meta-learning outperformed large language models like GPT-4o and Gemini 2.0 Flash on novel geometric transformation combinations.

• →Small transformer model with 5.7M parameters significantly outperformed state-of-the-art LLMs including o3-mini, GPT-4o, and Gemini 2.0 Flash on systematic generalization tasks.
• →Meta-learning for compositionality proves effective beyond linguistic tasks, extending successfully to abstract spatial reasoning problems.
• →Large language models show notable limitations in systematic generalization despite recent progress across various domains.
• →The small model performed on par with the winning 8B-parameter model from ARC prize 2024 that used test-time training.
• →Compositional-ARC dataset enables evaluation of models' ability to combine known geometric transformations in novel ways.