BilliardPhys-Bench: Benchmarking Physical Reasoning and Visual Dynamics of Multimodal LLMs

Researchers introduced BilliardPhys-Bench, a benchmark that tests multimodal AI models' ability to predict physical interactions in billiards simulations. The evaluation reveals that leading LLMs from OpenAI, Anthropic, Google, and Alibaba struggle with dynamic physics reasoning, exhibiting systematic failures including a 'stasis bias' where models default to predicting no interaction when physical outcomes become difficult to infer.

BilliardPhys-Bench addresses a critical gap in evaluating multimodal large language models—their capacity to reason about physical dynamics from visual input. While current MLLMs excel at static image classification, predicting object trajectories, collisions, and final states remains a fundamental weakness. The benchmark uses procedural generation to create diverse billiards scenarios with realistic friction and elastic collision mechanics, enabling systematic evaluation across complexity levels.

The research builds on growing recognition that visual understanding alone doesn't guarantee physical intuition. As AI systems become embedded in robotics, autonomous vehicles, and simulation tools, robust physical reasoning becomes commercially essential. The finding that performance degrades significantly as simulation duration increases and scene complexity grows suggests models lack true causal understanding of physics rather than simply pattern-matching visual features.

The 'stasis bias' discovery—where models predict inaction when outcomes become uncertain—carries particular significance for safety-critical applications. This reveals a failure mode that could propagate errors in real-world deployments. For developers integrating these models into physics-dependent applications, the benchmark provides concrete evidence of reliability limitations and highlights where additional training data or architectural changes are necessary.

These findings will likely accelerate research into incorporating explicit physical inductive biases into multimodal architectures. Companies developing foundation models for robotics and simulation face pressure to address these deficiencies. The standardized benchmark itself enables tracking progress and comparing approaches, establishing a foundation for measuring improvements in physical reasoning capabilities across future model iterations.

• →Leading multimodal LLMs from GPT, Claude, Gemini, and Qwen families demonstrate significant weaknesses in predicting physical dynamics and object interactions from images.
• →A systematic failure mode called 'stasis bias' causes models to predict no interaction when physical outcomes are difficult to infer, creating a safety concern for real-world applications.
• →Model performance degrades substantially as simulation time increases and scene complexity grows, indicating models lack true causal understanding of physics.
• →BilliardPhys-Bench provides a standardized evaluation framework using procedurally generated billiards scenarios to measure physical reasoning capabilities across multimodal architectures.
• →The research highlights the need for better physical inductive biases in future multimodal model designs, particularly for robotics and autonomous systems applications.