Mihir Prabhudesai^*,†, Aryan Satpathy^*,†, Yangmin Li^†, Zheyang Qin^†,
Nikash Bhardwaj, Amir Zadeh^λ, Chuan Li^λ, Katerina Fragkiadaki, Deepak Pathak

Carnegie Mellon University · ^λLambda

^*Project co-leads & Equal contribution · ^†Core contributors

teaser.mp4

We present SIM2REASON: a method for turning physics simulators into scalable generators of question–answer pairs to improve LLM reasoning, removing the need of human annotation in the data-generation pipeline. The core idea is to structure the randomization with a domain-specific language (DSL) and use it to procedurally generate reasoning problems, as illustrated in the examples above. LLMs finetuned on this synthetic data get zero-shot improvement on real world benchmarks such as International Physics Olympiad.

Abstract

We have witnessed remarkable advances in LLM reasoning capabilities with the advent of DeepSeek-R1. However, much of this progress has been fueled by the abundance of internet question–answer (QA) pairs—a major bottleneck going forward, since such data is limited in scale and concentrated mainly in domains like mathematics. In contrast, other sciences such as physics lack large-scale QA datasets to effectively train reasoning-capable models. In this work, we show that physics simulators can serve as a powerful alternative source of supervision for training LLMs for physical reasoning. We generate random scenes in physics engines, create synthetic question–answer pairs from simulated interactions, and train LLMs using reinforcement learning on this synthetic data. Our models exhibit zero-shot sim-to-real transfer to real-world physics benchmarks: for example, training solely on synthetic simulated data improves performance on IPhO (International Physics Olympiad) problems by 5-10 percentage points across model sizes. These results demonstrate that physics simulators can act as scalable data generators, enabling LLMs to acquire deep physical reasoning skills beyond the limitations of internet-scale QA data. Code available at: https://sim2reason.github.io/.

Installation

Create an environment for data generation with the following commands:

conda create -n pho_data python=3.11
conda activate pho_data
pip install bpy
pip install mujoco==3.3.4 ImageIO
pip install ipdb scipy tabulate pandas matplotlib
pip install hydra-core omegaconf
pip install tqdm wandb
pip install sympy math_verify
pip install transformers pyarrow

Create an environment for training with the following commands:

conda create -n pho_training python=3.12
conda activate pho_training
pip install vllm==0.8.5.post1
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.4.post1/flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whl
cd verl_v4
pip install -e .
pip install hydra-core==1.4.0.dev1 omegaconf==2.4.0.dev3
pip install math-verify==0.7.0
pip install latex2sympy2_extended==1.0.9
pip install antlr4-python3-runtime==4.11.0
pip install polars ipdb
pip install transformers==4.52.3
pip install setuptools==69.5.1

Preparing Data

Set path to store data and checkpoints:

export PHO_DATA=<path to folder>
export PHO_CHECKPOINT_DIR=<path to folder>
export DATA_VERSION=v1

Generate synthetic scenes by running the following command:

python -m sim.scene_generator scene_generation.num_scenes=1000 scene_generation=all data_version=$DATA_VERSION

Generate QA pairs and filter shortcut questions by running the following command:

python -m sim.qa_gen_rule data_version=$DATA_VERSION question_generation.num_generations_per_problem=30 numerical=True symbolic=True reverse=False
python -m sim.create_child_scenes data_version=$DATA_VERSION
python -m sim.qa_gen_rule data_version=$DATA_VERSION numerical=True symbolic=True reverse=False question_generation.build_child_scenes=True

Preprocess generated QA pairs into format suitable for verl by running the following command:

python -m sim.write_json data_version=$DATA_VERSION numerical=True symbolic=False reverse=False
python -m sim.write_json data_version=$DATA_VERSION numerical=False symbolic=True reverse=False
python -m llm.preprocess_json_to_parquet --json_names numerical_problems_all_train_without_shortcut.json numerical_problems_all_test_without_shortcut.json symbolic_problems_all_train_without_shortcut.json symbolic_problems_all_test_without_shortcut.json  --data_version "$DATA_VERSION" --no_extra_instruction

Upon successfully running the above, <PHO_DATA>/<DATA_VERSION> should be populated with two files: train_<DATA_VERSION>_rl.parquet and test_<DATA_VERSION>_rl.parquet.

Training

python -m verl_v4.recipe.dapo.main_dapo exps="[dapo_32b,syn_data,q2.5_14b,gspo,use_kl]"  trainer.n_gpus_per_node=8 trainer.val_before_train=False  sp_size=8 gen_tp=8 trainer.total_epochs=10

This command finetunes Qwen2.5 14B Instruct on the generated synthetic data using DAPO algorithm. Training info is logged in wandb by default in the project named verl, and checkpoints are saved at PHO_CHECKPOINT_DIR/<run_name>. By default, only latest global step checkpoint is saved, deleting the old checkpoints to save storage.

Pretrained Checkpints: We provide our pretrained checkpoints for various Qwen models in HuggingFace.

Evaluation

Evaluation requires first converting the checkpoint to HuggingFace format by running the following command:

export ACTOR_DIR=$PHO_CHECKPOINT_DIR/<run_name>/global_step_<step>/actor
python verl_v4/scripts/model_merger.py merge \
    --backend fsdp \
    --local_dir "$ACTOR_DIR" \
    --target_dir "$TARGET_DIR"

To evaluate the model's performance on IPhO, download the val set from HF link to PHO_DATA/ipho/ipho_numeric_validation_no_instruct.parquet and run:

python -m verl_v4.recipe.dapo.simple_eval \
 exps='[dapo_32b,log_all_reward,pass_at_n,simple_eval]' \
   model=qwen2.5-3b-instruct data.val_batch_size=null actor_rollout_ref.rollout.val_kwargs.n=8 \
  max_response_length=30000 trainer.n_gpus_per_node=8 model_name=${MODEL_PATH} \
  actor_rollout_ref.rollout.gpu_memory_utilization=0.85 max_model_len=30000 \
  data.val_files="[${PHO_DATA}/ipho/ipho_numeric_validation_no_instruct.parquet]" \
  data.train_files="[${PHO_DATA}/${DATA_VERSION}/train_${DATA_VERSION}_rl.parquet]"

Note for the 32B checkpoint: The released 32B checkpoint was trained with an older instruction prompt. To evaluate it correctly on IPhO, add use_legacy_prompt to the exps list:
python -m verl_v4.recipe.dapo.simple_eval \
  exps='[dapo_32b,log_all_reward,pass_at_n,simple_eval,use_legacy_prompt]' \
  model=qwen2.5-3b-instruct data.val_batch_size=null actor_rollout_ref.rollout.val_kwargs.n=8 \
  max_response_length=30000 trainer.n_gpus_per_node=8 model_name=${MODEL_PATH} \
  actor_rollout_ref.rollout.gpu_memory_utilization=0.85 max_model_len=30000 \
  data.val_files="[${PHO_DATA}/ipho/ipho_numeric_validation_no_instruct.parquet]" \
  data.train_files="[${PHO_DATA}/${DATA_VERSION}/train_${DATA_VERSION}_rl.parquet]"
All other benchmarks (JEEBench, OlympiadBench, PHYSICS) are unaffected and use the same eval command regardless of checkpoint.

To evaluate the model's performance on JEEBench, first download JEEBench by running setup_jeebench.sh, then run:

python -m verl_v4.recipe.dapo.simple_eval \
  exps='[dapo_32b,log_all_reward,pass_at_n,use_JEEBench]' \
  data.val_files="[${PHO_DATA}/JEEBench/dataset.json]" \
  model=qwen2.5-3b-instruct data.val_batch_size=1 actor_rollout_ref.rollout.val_kwargs.n=8 \
  max_response_length=32000 trainer.n_gpus_per_node=8 model_name=${MODEL_PATH} \
  actor_rollout_ref.rollout.gpu_memory_utilization=0.65 max_model_len=32000

To evaluate the model's performance on OlympiadBench, run:

python -m verl_v4.recipe.dapo.simple_eval \
  exps='[dapo_32b,log_all_reward,pass_at_n,use_olympiad_bench]' \
  data.val_files="[${PHO_DATA}/olympiad_bench/OlympiadBench_Dataset/data/OE_TO_physics_en_COMP.json,
  ${PHO_DATA}/olympiad_bench/OlympiadBench_Dataset/data/OE_TO_physics_zh_CEE.json]" \
  model=qwen2.5-3b-instruct data.val_batch_size=1 actor_rollout_ref.rollout.val_kwargs.n=1 \
  max_response_length=32000 trainer.n_gpus_per_node=8 model_name=${MODEL_PATH} \
  actor_rollout_ref.rollout.gpu_memory_utilization=0.65 max_model_len=32000

To evaluate the model's performance on PHYSICS, download the val set from HF link to PHO_DATA/PHYSICS/test.jsonl and run:

python -m verl_v4.recipe.dapo.simple_eval \
  exps='[dapo_32b,ipho_numeric_val,log_all_reward,pass_at_n,use_PHYSICS]' \
  model=qwen2.5-3b-instruct data.val_batch_size=1 actor_rollout_ref.rollout.val_kwargs.n=8 \
  max_response_length=32000 trainer.n_gpus_per_node=8 model_name=${MODEL_PATH} \
  actor_rollout_ref.rollout.gpu_memory_utilization=0.65 max_model_len=32000

Since the verifier used in PHYSICS is not opensource, we use Gemini 2.5 Flash as it is a strong verifier. Set the Google API Key in llm/utils/basic_utils.py line 138 before running this evaluation.

Citation

If you find this work useful in your research, please cite:

@article{prabhudesai2026solving,
  title={Solving Physics Olympiad via Reinforcement Learning on Physics Simulators},
  author={Prabhudesai, Mihir and Satpathy, Aryan and Li, Yangmin and Qin, Zheyang and Bhardwaj, Nikash and Zadeh, Amir and Li, Chuan and Fragkiadaki, Katerina and Pathak, Deepak},
  journal={arXiv preprint arXiv:2604.11805},
  year={2026}
}