LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models

Researchers introduce LAST, a framework that enhances multimodal large language models' spatial reasoning by integrating specialized vision tools through an interactive sandbox interface. The approach achieves ~20% performance improvements over baseline models and outperforms proprietary closed-source LLMs on spatial reasoning tasks by converting complex tool outputs into consumable hints for language models.

This research addresses a fundamental limitation in current multimodal AI systems: their struggle with precise spatial understanding and geometric reasoning. MLLMs frequently generate hallucinations when interpreting complex layouts, a problem that pure scaling hasn't solved because spatial constraints require structured priors rather than just more training data. The LAST framework takes a pragmatic approach by leveraging existing specialized vision models—which excel at segmentation, depth estimation, and object detection—rather than attempting to build all capabilities into a single model.

The innovation lies in the abstraction layer LAST-Box creates, which translates heterogeneous tool outputs into standardized multimodal hints that language models can efficiently consume. This bridges a critical gap: specialized vision tools produce low-level outputs like segmentation masks that aren't naturally interpretable by LLMs. By converting these into annotated images and textual descriptions, the framework enables effective tool-augmented reasoning.

The three-stage progressive training strategy represents thoughtful system design, guiding models through understanding outputs before attempting adaptive tool invocation. Achieving 20% performance gains and surpassing proprietary systems suggests the framework's broad applicability beyond spatial tasks—this pattern of tool augmentation could extend to other reasoning domains where specialized systems exist.

For the AI development community, this work validates the tool-augmented reasoning paradigm as a practical alternative to monolithic model scaling. It demonstrates that modular AI architectures combining specialized components often outperform attempts to consolidate all capabilities into single models, particularly for structured reasoning tasks.

• →LAST framework achieves ~20% performance gains by integrating specialized vision tools into multimodal LLM spatial reasoning workflows.
• →LAST-Box abstraction layer converts diverse tool outputs into multimodal hints that language models can directly consume and reason with.
• →Progressive three-stage training strategy enables models to transition from understanding tool outputs to autonomous, adaptive tool invocation.
• →Framework outperforms proprietary closed-source LLMs on spatial reasoning tasks despite being based on smaller MLLM backbones.
• →Research validates modular, tool-augmented AI architectures as superior to monolithic scaling for structured geometric reasoning.