This work was carried out primarily by Atharva Hans, with the help of Alex Alberts. Max and I are mostly observing.
Replicating a scientific paper is an important task. Every researcher has to grind through this process, often many times. I have done it repeatedly while creating educational activities for my courses. I also ask graduate students to do it to develop their mathematical and coding skills. Every time we encounter a new dataset to analyze, we typically start by trying an existing method. And every time we develop a new method, we compare it against the state of the art.
Sometimes, replicating a paper is relatively straightforward, especially if the authors carefully include all required information about parameters and hyperparameters and also provide the code. Even then, the process takes time. First, you have to read and understand the paper. Depending on your familiarity with the literature, this may take anywhere from a few days to an entire month. You may also need to set up your computational environment, download and understand the data, run cluster jobs, move data around, generate figures, and so on.
More often than not, however, something is missing. The code may not be available. You may have to write it yourself, debug it, and verify it with synthetic examples. Important details may also be absent from the paper. For example, the authors may say that they performed “hyperparameter optimization” without specifying how. Or the paper may rely on methods you are not familiar with, forcing you to read additional papers before you can even begin implementing the approach.
Using AI agents to replicate papers is, therefore, an intriguing use case. Tools like Codex, Claude Code, and OpenClaw have already reached a stage where they can attempt such tasks. We have started putting them to the test.
We have an old Mac Pro machine with Homebrew installed. We also installed LaTeX for generating reports. We have a ChatGPT Pro subscription ($200/month). We run the Codex app for macOS with GPT-5.3-Codex using “Medium” thinking, and we give it “Full access.”
We set up SSH keys so the system can access our lab’s GitHub repository (PredictiveScienceLab). We launch the Codex app inside an existing repository. Thankfully, Codex has not yet escaped and started deleting our prior work…
We configured SSH keys that allow Codex to access Purdue’s high-performance computing clusters (Gautschi and Bell) to run code on both GPUs and CPUs. We created a cluster skill that teaches it how to transfer data, load modules, submit SLURM jobs, and monitor them. Here are the first few line of the kill:
Note that the skill also has some scripts that it can call.
You may have guessed that the prompt “replicate this paper” doesn’t quite do the trick. It takes some prompt engineering to obtain useful results. We have converged on a reasonably good version, part of which you can see below (it is not perfect; we will discuss its limitations in a future post).
We experimented with both PDF versions of papers and with the LaTeX source code. Empirically, we observed that Codex cannot perfectly extract the mathematics from PDFs. It often introduces gaps or errors in the math, requiring multiple iterations to fix. The LaTeX source works much better. All the experiments below use the LaTeX source.
I will show results from three different papers. One lies clearly inside the training distribution of GPT-5.3-Codex. One (or possibly two) lies near the edge of that distribution. And one lies outside the distribution.
The final paper has already been accepted for publication. We have the code, but it has not yet been made publicly available.
The first example is SINDy. The mathematical idea is straightforward but powerful. The basic method uses observations of the state and its rate of change in a dynamical system. It attempts to fit the so-called vector field using a predefined set of basis functions. In other words, it casts system identification as a regression problem and adds L1 regularization to promote a sparse representation of the vector field.
You can learn more about it from the excellent lectures of Steve Brunton, the creator of the method.
The method has been applied to many different problems. It appears in thousands of papers and numerous textbooks. There are also many coded examples available online, including some in my own online textbook. We can therefore say that this is a methodological paper well inside the distribution that GPT-5.3-Codex likely saw during training.
In the second example, we ask our AI agent to replicate Alex Alberts’ paper on physics-informed information field theory. The method estimates the unobserved state of a physical field (e.g., temperature or pressure) as a function of space and time from sparse, nonlinear measurements. It incorporates prior knowledge of physics—such as PDE constraints or an energy residual—to emphasize physically plausible fields. It uses Feynman path integrals to define a physics-informed prior measure over the function space of physical fields. The method is also capable of estimating unknown physical parameters and detecting model-form uncertainty. You can watch my video to learn more.
The code for the original implementation is written in C++. We did not provide this code to the AI agent, although it could potentially locate it if it searched for it.
The method relies on a custom nested stochastic gradient Langevin dynamics algorithm. Stochastic gradient Langevin dynamics avoids the need to compute expensive spatiotemporal integrals. The “nested” component was inspired by contrastive divergence, a technique developed by Jeffrey Hinton. This nesting is necessary to handle an intractable normalization constant.
I would say this paper lies within the training distribution of GPT-5.3-Codex, but it is a rare event within it.
The final example is another paper by Alex Alberts. The method endows deep neural networks with priors that resemble Gaussian processes, thereby enabling Bayesian deep learning and, consequently, uncertainty quantification in predictions. The approach again uses Feynman path integrals to construct these priors, along with a carefully designed version of stochastic gradient Langevin dynamics to characterize the parameters.
This paper is clearly outside the training distribution of GPT-5.3-Codex. It has never been seen before, and there are no closely related examples in the literature.
You can find the results in this GitHub repo. Each case study is in a separate folder. The structure is almost the same. For example, the SINDy structure is like this:
If you go and look inside the “raw_output” folder, you will see everything that Codex did. It is instructive to read the “todo.md” file, where it tracks its progress. You can find the code in the “src” directory and the LaTeX for the reports in the “report” directory.
For your convenience, here are the three PDF reports:
All replications, except BNN (the only one out of distribution), succeeded. We have clear evidence that AI agents can replicate computational work to some extent.
We are actively using a broader range of methods and investigating why Codex failed to replicate the previous paper. Was it because it was outside of distribution? Should we have waited longer? Should we improve our verification mechanisms?
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.