Solving Physics Olympiad via Reinforcement Learning on Physics Simulators

Researchers demonstrate that physics simulators can generate synthetic training data for large language models, enabling them to learn physical reasoning without relying on scarce internet QA pairs. Models trained on simulated data show 5-10 percentage point improvements on International Physics Olympiad problems, suggesting simulators offer a scalable alternative for domain-specific AI training.

This research addresses a critical bottleneck in LLM development: the scarcity of high-quality training data in specialized domains beyond mathematics. While DeepSeek-R1 and similar models have advanced reasoning capabilities, their progress depends heavily on internet-scale question-answer datasets concentrated in a few domains. Physics, despite its importance, lacks comparable data resources, limiting model performance on complex reasoning tasks.

The study demonstrates that physics engines can synthetically generate unlimited training examples by simulating physical scenarios and extracting question-answer pairs from simulated interactions. Using reinforcement learning on this synthetic data, the researchers achieved meaningful performance gains on IPhO benchmarks—a proxy for genuine physical reasoning ability. The zero-shot transfer from simulation to real-world benchmarks validates that models internalize generalizable physics principles rather than memorizing dataset patterns.

This approach has significant implications for AI development beyond physics. It suggests that domain-specific simulators could unlock reasoning capabilities in chemistry, biology, engineering, and other sciences facing data scarcity. Organizations can now generate effectively unlimited training data for specialized domains, reducing dependence on human-curated datasets and accelerating capability development in underserved fields.

The framework also enables continuous improvement: as simulators become more sophisticated, training data quality automatically improves. However, simulation fidelity remains critical—sim-to-real gaps could limit transfer in some domains. Going forward, watch for applications of this technique in other scientific domains and whether industry adoption accelerates LLM development in specialized fields.

• →Physics simulators generate unlimited synthetic training data for LLMs, bypassing internet QA dataset scarcity in specialized domains.
• →Models trained solely on simulated data improved International Physics Olympiad performance by 5-10 percentage points across model sizes.
• →Zero-shot sim-to-real transfer demonstrates that LLMs learn generalizable physical reasoning principles from synthetic data.
• →This approach potentially unlocks reasoning capabilities in chemistry, biology, and engineering where domain-specific QA data is limited.
• →Continuous simulator improvement automatically enhances training data quality without requiring additional human annotation.