Background

I have been working on this idea for our course AI Engineering

Causal ML has a lot of potential in itself - but I see it more as a combination of AI (llms) and causal. In this post, I explain a relatively simple approach to use LLMs and causal ML together - in an easy to understand manner. I also explain the significance of causal ML in comparison to traditional machine learning techniques.

By keeping it simple - I think we could create real solutions to complex problems that combine these two approaches.

The overall problem statement could be formulated as follows:

Helping secondary school students improve their maths performance. Goal: Find out what actually causes better results (not just what’s correlated).

Think of a causal machine learning system as a very careful detective:

It doesn’t just notice patterns (“these two things move together”), it tries to figure out what actually causes what.

We build this system in two layers:

• Questionnaire → to collect structured information from people on the ground

• LLM (like ChatGPT) → to help design questions, interpret answers, and support the causal reasoning

Process

1. Define the decision question

“Which actions actually improve students’ maths outcomes?”

E.g.

• Extra after-school tutoring

• Access to an online learning app

• Smaller class size

• Teacher training in new methods

• Parental homework support

This is a decision question:

“If we invest in X, does it cause better maths results?”

You can ask an LLM: “I want to reduce maths failure rates in secondary school. Suggest 5–7 concrete decision questions a policymaker might ask.”

2. Map possible causes and influences (in plain language)

We brainstorm what might affect maths outcomes:

We can use the LLM to prompt ideas like: “List factors that might influence secondary school maths performance, and separate them into:

• things we can change,

• background/context,

• possible confounders.

This becomes your design checklist

For example, in the tutoring example:

Possible causes (levers we can change)

• Extra tutoring (yes/no, hours per week)

• Use of a maths learning app at home or school

• Teacher training in new teaching methods

• Frequency of formative feedback (quizzes, practice tests)

• Parental support with homework

Context factors (not easily changed but important)

• Student’s prior maths ability

• Socio-economic background

• School resources (number of teachers, computers)

• Class size

• Language spoken at home

Confusing influences (confounders)

• Highly motivated students might Seek extra tutoring, Use apps more and Have higher scores anyway

• Schools with more resources might offer Smaller classes, Better-trained teachers and

• Better results

We need to measure these to compare fairly.

3. Turn the causal story into a questionnaire

Now we convert these ideas into questions for students, parents, and teachers.

3.1 Outcome questions (what we’re trying to improve)

• “What was your grade in the last maths exam?” (letter or numeric)

• “On average, how confident do you feel in maths?” (1–5 scale)

• (From school records) pass/fail, score out of 100

3.2 Questions about interventions (potential causes)

a) To students:

• “In the last 3 months, did you attend extra maths tutoring?” No/ Yes, once a week / Yes, 2–3 times a week / Yes, more than 3 times a week

• “Do you use a maths learning app (like Khan Academy, etc.)?” - Never / 1–2 times a week / 3–5 times a week /Nearly every day

b) To teachers:

• “Have you received specific training in new maths teaching methods in the last 2 years?” [Yes/No]

• “How often do you give short practice quizzes?” - Rarely / Once a month / Once a week /More than once a week

c) To parents:

• “How often do you help your child with maths homework?” Never / 1 day/week / 2–3 days/week / 4+ days/week

3.3 Questions about background / confounders

a) To students/parents:

• “Did you usually pass maths in previous years?” (Yes/No)

• “How many books are there at home (roughly)?” (as a rough proxy for home learning environment)

• “How many siblings do you live with?”

• “What language do you mostly speak at home?”

b) From school data:

• Class size

• Teacher years of experience

• School resource index (computers per student, etc.)

3.4 Open-ended questions

• To students: “What makes maths easier or harder for you?”

• To teachers: “What do you think most helps students improve in maths?”

• To parents: “What stops you from helping more with homework (if anything)?”

Where LLM helps

• Drafting clear, age-appropriate questions

• Checking for leading or confusing wording

• Translating into simpler English or other languages

4. Collect the data

• Run the survey across many classes/schools

• Merge data: Questionnaire data (students, parents, teachers)

  and School records (exam scores, class size, etc.)

Each student becomes one row in a table, with columns like:

• extra_tutoring_hours

• uses_app_frequency

• teacher_trained

• parent_homework_help

• prior_maths_result

• class_size

• exam_score

5. Write down the causal story in simple terms

We now say explicitly:

“We believe that extra tutoring, use of a maths app, and teacher training can cause better maths results. But we also know that prior ability, family background, and school resources might influence both the chances of getting these interventions and exam scores.”

The key idea:

“We want like-for-like comparisons: students with similar background and ability, but different exposure to, say, extra tutoring.”

Where LLM helps Ask: “Given this context, describe a causal diagram in words, listing arrows between key variables.”

This helps you and stakeholders reason clearly before any statistics.

6. Causal machine learning (explained without maths)

Now the causal ML system uses the data to answer questions like:

“If a student starts attending weekly tutoring, how much will their maths score likely improve, on average?”

What the system does in plain terms

Finds similar students

• Same prior maths level

• Similar socio-economic background

• Similar class size / school resources

Looks at who got the intervention

• Group A: students who began extra tutoring

• Group B: very similar students who didn’t

Compares outcomes

• Average exam score in Group A vs Group B

• Difference = estimated effect of tutoring, after considering background factors

Repeats for other interventions

• Using a maths app

• Teacher training

• More frequent quizzes

• Parental homework help

So we can say things like:

“For students with similar background and prior ability, those who attended weekly tutoring scored on average 6 points higher out of 100.”

“Using the maths app at least 3 times a week is associated with 4 points higher, controlling for other factors.”

This is causal ML in friendly language:

“Carefully comparing similar students to estimate what difference an intervention makes.”

7. How LLMs complement the causal analysis

LLMs don’t do the statistics, but they are powerful assistants around it.

7.1 Explaining results to non-experts

Technical output might say:

“Average treatment effect of tutoring = +6.2 points (95% CI 3.5–8.9).”

You give this plus context to an LLM and ask:

“Explain these findings for head teachers and parents in simple language.”

It can produce:

“For students similar to those in this study, starting weekly maths tutoring raised exam scores by about 6 points out of 100, on average. Some improved more, some less, but the evidence suggests tutoring meaningfully helps.”

7.2 Analysing open-ended answers

Feed the free-text responses to an LLM and ask it to:

Group common themes:

“Maths anxiety”

“Lack of quiet study space”

“Internet/data problems”

“Parents don’t understand the maths”

Provide quotes to illustrate each theme

Link these to interventions

Example narrative:

“Many students mention they feel anxious during tests. This suggests that tutoring might work even better when combined with simple test-anxiety strategies.”

7.3 Generating “what-if” stories for policy

Once you know the estimated effects, you can ask:

“Based on our findings, write a 1-page policy brief for the school board about whether to invest more in tutoring, apps, or teacher training, using plain English.”

The LLM will:

• Weave numbers + context into a story

• Present trade-offs (e.g., cost, feasibility)

• Suggest phased pilots (“start with Year 9, evaluate, then scale”)

7.4 Checking assumptions and limitations

Ask:

“Given our causal setup (tutoring, apps, teacher training, background factors), what assumptions might fail, and what biases could affect our conclusions?”

The LLM might highlight:

• Some students self-select into tutoring because they are more motivated

• Tutors might differ in quality

• Some schools may have unmeasured advantages

This helps you design better follow-up studies or randomised pilots.

8. One-paragraph story you can tell stakeholders

“We surveyed students, parents, and teachers to understand who gets extra tutoring, uses maths apps, and receives trained teaching—and we also recorded their backgrounds and exam scores. Then, using causal machine learning, we compared students who are similar in background and prior ability but differ in these interventions. This allowed us to estimate how much tutoring, app use, or teacher training actually causes improvement in maths scores. Large language models (like ChatGPT) helped us design clear questionnaires, summarise people’s written comments, and explain statistical results in plain language. Together, this system turns everyday school experiences into reliable evidence about what really helps students learn maths.”

Causal vs non causal

We can also compare causal and non causal approaches.

This is a very important question—because it makes clear why you bothered with causal ML at all.

We can explain this idea using the same example - Helping secondary school students improve maths performance” problem.

Think of non-causal techniques as answering: “What usually goes together?”

and causal techniques as answering: “What will happen if we change something?”

Non-Causal Techniques for Improving Maths Performance

1 Descriptive Statistics

(Averages, percentages, charts)

What it looks like

You might calculate:

• Average maths score of students who attend tutoring

• Average maths score of students who don’t

• Percentage of high-scoring students using a maths app
  

Example finding:

“Students who attend tutoring score 15 points higher on average.”

Why it’s attractive

• Simple

• Easy to explain

• Fast to compute

• Great for summarising data
  

Key limitation (non-causal)

This does not tell you why the difference exists.

Possible explanations:

• Tutored students were already stronger

• Their parents are more involved

• Their schools have better resources

Core problem:
Descriptive stats confuse cause and coincidence.

They answer: “What does the world look like?”

But not:“What would change if we intervene?”

2️Correlation Analysis

(“X is correlated with Y”)

What it looks like

You compute correlations such as:

• App usage ↔ exam score

• Homework help ↔ confidence

• Class size ↔ performance

Example finding:

“Use of maths apps is strongly correlated with higher scores.”

Why it’s attractive

• Slightly more “scientific” than averages

• Identifies interesting relationships

• Good for hypothesis generation
  

Key limitation (non-causal)

Correlation does not tell you direction or mechanism.

For example:

• Do apps improve maths?

• Or do good maths students enjoy apps?

• Or do better schools provide both apps and good teaching?
  

Core problem:
Correlation cannot distinguish:

• Cause → effect

• Effect → cause

• Shared underlying cause

So it cannot support policy decisions.

3️Predictive Machine Learning

(Regression, random forests, neural networks)

What it looks like

You train a model to predict maths scores using inputs like:

• Tutoring hours

• App usage

• Parental help

• Class size

• Past performance

The model predicts:

“This student will likely score 72/100.”

Why it’s attractive

• High accuracy

• Handles many variables

• Feels “advanced” and modern

• Useful for early warning systems

Key limitation (non-causal)

Predictive models learn patterns in existing data, not consequences of change.

If you ask:

“What if we give this student tutoring?”

The model cannot answer reliably because:

• It has never seen the counterfactual (what would have happened otherwise)

• It treats tutoring as just another feature, not an intervention

Core problem:
Prediction ≠ intervention.

These models answer: “What is likely to happen?”

Not: “What will happen because we act?”

4️Feature Importance & Explainable AI

(SHAP values, feature rankings)

What it looks like

You might see outputs like:

“Top predictors of maths success:

1. Prior maths score

2. App usage

3. Parental education

4. Tutoring”

Why it’s attractive

• Feels interpretable

• Shows “what matters most”

• Useful for communication

Key limitation (non-causal)

Important predictors are not necessarily good levers.

Example:

• Prior maths score is the strongest predictor → But you can’t change it directly.

• Parental education predicts success → But schools can’t intervene on it.

Core problem: Feature importance ranks predictors, not causes you can act on.

5️Clustering & Segmentation

(Grouping students)

What it looks like

You cluster students into groups:

• “High achievers”

• “Struggling but motivated”

• “Low engagement”

• “High anxiety”
  

Why it’s attractive

• Helps tailor support

• Useful for personalised learning

• Good for targeting resources
  

Key limitation (non-causal)

Clusters describe who is similar, not what will help them.

Example:

• You identify a “low engagement” group

• But you still don’t know:

  • Should you give them tutoring?

  • Apps?

  • Counselling?

  • Teacher changes?

Core problem: Segmentation tells you who, not what to do.

Why These Methods Fail for Decisions

All non-causal methods share one limitation:

They describe the world as it is — not the world after we intervene.

In decision-making, the real question is always:

“If we do X instead of Y, what will change?”

Non-causal techniques:

• Cannot simulate “what if”

• Cannot separate cause from selection

• Cannot estimate true impact

• Cannot support cost-benefit trade-offs

Why Causal ML Is Different

Causal ML is built specifically to answer:

“What will happen if we intervene?”

For example:

• If we add weekly tutoring, what happens to scores?

• If we train teachers, does that cause improvement?

• If we increase app usage, does it really help?

This is why the approach:

Questionnaire + causal thinking + LLM support is a practical architecture for real-world decision-making.

“Traditional analytics can tell us who is doing well and who is struggling, but only causal analysis can tell us which actions will actually improve student outcomes.”

The intriguing question

For me, the intriguing question is as follows

If we create the dataset in step 4 (which we have to in any case)

ex

• Run the survey across many classes/schools

• Merge data: Questionnaire data (students, parents, teachers)

  and School records (exam scores, class size, etc.)

Each student becomes one row in a table, with columns like:

• extra_tutoring_hours

• uses_app_frequency

• teacher_trained

• parent_homework_help

• prior_maths_result

• class_size

• exam_score

Then, could we directly prompt against that dataset and get a causal analysis? without going through a causal system (ex Python coding)?

Its worth exploring!

Our course at Oxford is AI Engineering

Image source: unknown