Can an LLM agent discover a causal law the way a scientist would — by experimenting?

⚠️ v0.0.2 — First Public Release. Interfaces and dataset formats may change. Please open an issue if you encounter problems.

The core question

A real scientist doesn't look up how the world works. They intervene on it, watch what changes, and revise a theory until it transfers to a case they've never seen. Most LLM benchmarks test retrieval — can the model recall a known fact? CausaLab tests discovery: can an agent run experiments, recover the hidden causal mechanism, and apply it to a novel case from scratch?

Each episode hides a freshly sampled structural causal model (SCM) inside a synthetic crystal reactor. The agent manipulates variables, observes outcomes, and must both predict a held-out measurement and recover the causal graph that explains it. You can't win by reciting memorized facts.

Three findings

🎯 Prediction ≠ Understanding

GPT-5.2-high reaches 92% task accuracy on 6-node graphs — but only 0.47 all-edge F₁.

Right number. Wrong graph.

🔬 The act of experimenting matters

Handing the agent perfect pre-collected data ("Golden") boosts prediction but not mechanism recovery.

The choice of experiment carries structural signal.

⏱️ Agents stop too soon

Win or lose, runs leave ~half their intervention budget unused.

One "check theory vs. evidence" step lifts 4-node accuracy: 48% → 60%.

How It Works

┌─────────────────────────────────────────────────────────────────┐
│  Hidden SCM  →  prior records  →  agent observes               │
│                                                                  │
│  Agent intervenes on manipulator crystal                        │
│  ↓  (repeat until budget exhausted)                             │
│  Agent emits DSL causal hypothesis at each step                 │
│  ↓                                                               │
│  Final prediction: reactor crystal's hidden frequency           │
│                                                                  │
│  Scoring: prediction accuracy  +  graph/equation F₁             │
└─────────────────────────────────────────────────────────────────┘

A hidden SCM governs both a manipulator crystal (the agent can poke it) and a reactor crystal (held out). The agent must discover the shared causal law through experimentation, then transfer it. We score two things: whether the agent got the right answer, and whether it recovered the right mechanism.

Repository structure

CausaLab-Benchmark/
│
├── discoveryworld/              # Reactor Lab environment (DiscoveryWorld fork)
│   └── scenarios/reactor_lab.py #   ← the CausaLab task definition
│
├── agents/recoma/               # ReAct / Recoma agent
│   ├── run_recoma.py            #   ← main agent runner
│   ├── react_controller.py      #   ← controller loop
│   └── prompts/                 #   ← DSL & non-DSL prompt variants
│
├── scripts/
│   ├── experiments/             # Launch scripts, summarizer, monitor
│   ├── vis_backend/             # Trajectory visualization server
│   └── vis_frontend/            # Visualization UI
│
├── causalab_reeval/             # ReEval scoring module
│
├── release/causalab_dataset/    # Synthetic graph configs — 950 records
│   └── data/                    #   ← 19 JSONL suites, 3–7 nodes
│
└── examples/sample_runs/        # Bundled sample trajectory for offline checks

Installation

Requires Python 3.10

conda create -n causalab python=3.10 -y
conda activate causalab
pip install --upgrade pip
pip install --use-pep517 -r agents/requirements.txt
pip install -e .

For API-backed runs:

export OPENAI_API_KEY=sk-...
export OPENAI_API_BASE=https://api.openai.com/v1   # optional override

Offline smoke checks do not require an API key.

Quick Start

1 — Dry-run (no API calls)

SKIP_CONDA_ACTIVATE=1 DRY_RUN=1 GRAPH_LIMIT=1 \
  bash scripts/experiments/run_react_simple-mem_freqparent_4nodes_main.sh

2 — ReEval on the bundled sample

python -m causalab_reeval.run_lightweight_reeval \
  --source-only --limit 1 \
  --output-dir output_dir/reeval_sample \
  examples/sample_runs/react_simple-mem

3 — Trajectory visualization

python scripts/vis_backend/visualization_server.py
# → open http://127.0.0.1:5001

Running Experiments

All scripts live in scripts/experiments/ and resolve paths from the repo root. Outputs go to output_dir/.

Variable	Effect
`DRY_RUN=1`	Print planned jobs, no API calls
`GRAPH_LIMIT=N`	Restrict to first N graph configs
`SEEDS_PER_GRAPH=N`	Seeds per graph
`BATCH_SIZE=N`	Concurrent local jobs
`SKIP_CONDA_ACTIVATE=1`	Use current Python environment

Experiment scripts reference

# ── Main runs ─────────────────────────────────────────────────────
bash scripts/experiments/run_react_simple-mem_parallel.sh          # GPT-5-mini 4/6-node
bash scripts/experiments/run_react_simple-mem_scaling_gpt52.sh     # GPT-5.2 scaling
# ── Scaling suites ────────────────────────────────────────────────
bash scripts/experiments/run_react_simple-mem_scaling_suite.sh
bash scripts/experiments/run_react_simple-mem_scaling_suite_6nodes.sh
# ── Formula variants ──────────────────────────────────────────────
bash scripts/experiments/run_react_simple-mem_4nodes_quad.sh
bash scripts/experiments/run_react_simple-mem_4nodes_quad_hard.sh
# ── Hidden-variable suites ────────────────────────────────────────
bash scripts/experiments/run_hidden_variants_full.sh
bash scripts/experiments/run_hidden_frequency_node_priority_full.sh
# ── Oracle / FreqParent / Golden follow-ups ───────────────────────
bash scripts/experiments/run_react_simple-mem_oracle_main_suite.sh
bash scripts/experiments/run_react_simple-mem_freqparent_4nodes_main.sh
bash scripts/experiments/run_react_simple-mem_golden_4nodes_main.sh

Summarizing results

python scripts/experiments/summarize_completion_flags.py \
  output_dir/react_simple-mem/obs_CausalFreqParent_gpt-5-mini/4nodes_main/<run_id> \
  --json-out output_dir/summary.json \
  --csv-out  output_dir/flags.csv

Dataset

release/causalab_dataset/ contains 950 synthetic causal graph configurations across 19 JSONL suites (3–7 nodes; standard, quadratic, hidden-variable, FreqParent, and Golden variants). Ships with Croissant metadata and SHA-256 checksums.

import json
from pathlib import Path
configs = [
    json.loads(line)
    for line in Path("release/causalab_dataset/data/4nodes.jsonl").read_text().splitlines()
]
print(len(configs), configs[0]["graph_id"])
# → 50  4nodes_0

See release/causalab_dataset/README.md for the full schema and field reference.

Citation

@article{zhang2025causalab,
  title     = {CausaLab: Interactive Causal Discovery Toward AI Scientists},
  author    = {Zhang, Dylan (Shizhuo) and Yang, Junlin and Song, Xiangchen
               and Dai, Qirun and Liu, Xiao and Chen, Yuen and Vashishtha, Aniket
               and Shi, Jing and Tan, Chenhao and Peng, Hao},
  journal   = {arXiv preprint arXiv:2605.26029},
  year      = {2025},
  url       = {https://arxiv.org/abs/2605.26029},
}

Authors

Dylan Zhang · dylanzsz.github.io · @dylan_works_ · shizhuo2@illinois.edu

With Junlin Yang, Xiangchen Song, Qirun Dai, Xiao Liu, Yuen Chen, Aniket Vashishtha, Jing Shi, Chenhao Tan, and Hao Peng — across UIUC, Tsinghua, CMU, University of Chicago, and Adobe.