MaD Physics: Evaluating information seeking under constraints in physical environments

Researchers introduce MaD Physics, a benchmark for evaluating AI agents' ability to conduct scientific discovery under realistic resource constraints. The benchmark tests agents' capacity to make informative measurements within budget limits and infer underlying physical laws, using altered physics environments to prevent reliance on training data.

MaD Physics addresses a significant gap in AI evaluation methodologies by introducing constrained experimental design into scientific discovery benchmarking. Traditional benchmarks focus either on static knowledge reasoning or unconstrained tasks, failing to capture the real-world constraints that govern scientific inquiry. This work matters because it pushes AI evaluation toward practical scientific applications where measurement quality, quantity, and cost are fundamental tradeoffs.

The benchmark's design reflects genuine scientific challenges: researchers must allocate limited measurement budgets, plan exploration strategically, and draw conclusions from incomplete data. By incorporating altered physical laws across three distinct environments, the researchers prevent models from simply retrieving memorized physics, forcing genuine reasoning about experimental design and data interpretation.

The evaluation of Gemini models (2.5 Flash Lite through 3 Flash) reveals that current large language models struggle with structured exploration and systematic data collection. These capabilities are essential for autonomous scientific discovery systems that could accelerate research in materials science, drug discovery, and physics exploration. The identified shortcomings—particularly in planning under constraints and sequential measurement optimization—represent concrete engineering challenges for AI developers building scientific reasoning systems.

Beyond immediate AI development, this benchmark establishes evaluation standards that research institutions and AI companies will likely adopt. Success on MaD Physics could differentiate models marketed for scientific applications. The work signals growing maturity in AI evaluation, moving from general reasoning benchmarks toward domain-specific, constraint-aware assessment that reflects real operational requirements.

• →MaD Physics introduces the first benchmark specifically designed to evaluate AI agents' scientific discovery capabilities under realistic measurement budgets and quality constraints.
• →Current Gemini models exhibit significant weaknesses in structured exploration and data collection planning despite general reasoning capabilities.
• →The benchmark uses altered physical laws to prevent knowledge contamination and ensure agents perform genuine scientific reasoning rather than retrieving training data.
• →Successful performance on constrained scientific discovery could become a key differentiator for AI models targeting research and scientific applications.
• →This work establishes evaluation standards that will likely influence future development of autonomous AI systems for experimental science and research acceleration.