Feedback Machines II: Using AI to Generate Feedback on Your Research

This post is about how to use Claude Code (or OpenAI Codex) to generate detailed, structured feedback on your research. I have written before about why feedback matters and how AI changes access to it. Here, I want to share the specific tools I have built and how I use them. This post will describe four different tools: three tools for reviewing writing, and one tool that checks your code and compares it to what is written in the paper. Here is the Github folder with these skills, they are free to download and use.

I am not trying to build a system that writes papers from scratch. I am trying to get good, structured feedback on specific parts of the research process. The future will tell whether this is a good career move. Maybe I could have many more papers ready to submit to journals if I automated more, but I am not really that interested in automating everything. I worry about using AI tools too much and about losing something in the process if I automate too much. I am interested in learning and discovering things in the world, not just in producing papers for journals.

I think these tools are tremendously helpful and that they can help make my research better. At the end, I briefly discuss how I created them. I hope they help you!

I currently have three writing feedback tools. Turns out that lots of writing in academia is relatively formulaic and that AI tools will have lots of implicit knowledge of how such writing should look. That makes them very useful for providing feedback. I am sure there are lots of other instances where this is true, and if you find one, just ask Claude to adapt one of these tools I have built to your own purpose. Imagine how good our writing will be in the future!

The pre-submission paper review (/review-paper) is the one I use most. It runs six specialized agents in parallel and consolidates their findings into a single structured report. Each agent focuses on a different dimension: spelling, grammar, and academic style; internal consistency and cross-reference verification; unsupported causal claims and identification integrity; mathematics and notation; tables, figures, and their documentation; and finally, a contribution evaluation that plays the role of an adversarial associate editor. For that last agent, you can specify a target journal, and the agent will adopt the editorial standards of that venue. I’m not actually sure this works well, but it’s perhaps helpful to set the bar somewhere. The output is organized by priority, with issues tagged as critical, major, or minor.

At some point in the last week, I was working on a pre-analysis plan for an experiment, and I realized that a similar framework for feedback could be useful. I had Claude adapt the paper review to a pre-analysis plan review instead, which it did pretty well. The tool checks whether hypotheses are operationally defined, whether the statistical analysis plan is sufficiently binding to prevent post hoc flexibility, whether power calculations are realistic, and whether the identification assumptions are clearly stated. It also evaluates whether the PAP meets the standards of the specific registration platform you are targeting — AEA, EGAP, OSF, and others.

Then I was writing a grant application, and I had Claude create a skill for that, too. This tool runs agents focused on clarity and compliance signals, internal consistency across the narrative and budget, significance and innovation claims, methodological feasibility, budget and timeline credibility, and a final adversarial panel reviewer who evaluates whether the proposal is competitive for the target funder.

One thing I have realized is that economists very rarely get feedback on their code. There is naturally a hesitancy to post code publicly, given how long it takes to publish a paper in economics. Once the paper is accepted, you have to provide a replication package for certain journals nowadays, but I think I am right in saying that these mostly check if the code will run. This means that both mistakes and fraud can naturally creep in, and it is very difficult for reviewers to spot these (they don’t see the code, only the results!). Some people are doing great work replicating papers and finding errors, but finding errors in code requires that you have the replication package, which is typically only available after publication. This article in Nature finds that about 25% of the evaluated articles in economics and social sciences contained non‑trivial coding errors. That seems like a lot!

This poses two distinct problems. First, we rarely get feedback on code. At the very least, I rarely got feedback on my code. Obviously, AI tools can help with that. But second, and more importantly, it is not always clear that the code matches what is in the paper. So, I thought I could build a tool to try to solve some of these issues.

The “review-paper-code” checks whether the code actually implements what the paper describes. It reads your LaTeX files and your code files (Stata, R, or Python) and cross-checks them: are all the tables in the paper produced by the code? Does the sample construction in the code match the sample description in the data section? Does the paper say IV, but the code runs OLS? For anyone submitting to a journal that requires a replication package, which is increasingly most journals, this is hopefully useful. It also surfaces reproducibility problems: hardcoded file paths, missing random seeds, and no master script that runs the analysis in order. This skill is a bit of a work-in-progress, so please let me know if you have any ideas for improving it!

If you have Claude Code installed, these tools can be run from the command line in your project directory. The paper review, PAP review, and grant review all auto-detect your main document, so in most cases, you just type the command and the review runs. For the paper review, you can optionally specify a target journal: /review-paper AER will apply AER-specific standards. The paper-code correspondence check also auto-detects your project structure.

The output is saved as a markdown file in your project directory. For the paper review, this is a pre-submission report organized with an overall assessment, a preliminary recommendation, and a prioritized list of issues. I typically read through the critical items first, check whether I agree, and then work through the major items. The minor items I usually address in a final pass before submission.

Let’s end with a confession. I did not write the underlying prompts for these tools myself. I described to Claude what I wanted and asked it to build the skill. I started by asking for a review-paper skill covering different parts (consistency, spelling, contribution, etc), since that is how I tended to use AI tools in the first place. This turned out to work very well, in that I got some really nice feedback on my own papers.

Claude Code works by letting you write custom “skills” — essentially prompts that the assistant executes when you call them from the command line. You can build a skill that runs multiple AI agents in parallel, each with a specialized role, and then consolidates their findings into a single report. The resulting output is much richer than what you get from simply pasting a draft into a chat window and asking for feedback.

To make the process better, I asked Claude to launch a separate agent whose only job was to criticize the skill and identify gaps: places where the coverage was too shallow, where the review criteria were too vague, where a specialist reviewer would ask something else, or where the skill was just not working. Based on that criticism, I revised the skills. The process took a few iterations, but the result is considerably sharper than anything I would have produced on my own.

A quick note on launching agents: this is not complicated. Launching an AI agent is as simple as telling Claude “Launch another agent to evaluate and criticize XXX”. There is nothing magical or technical involved in this. Claude itself fills in the prompt for the agent, which is pretty neat. This recursive process of building tools with AI, then using AI to critique those tools, is, I think, underused. I would recommend trying it out.

These tools are useful, but they are not a substitute for real feedback from real economists. They are good at catching inconsistencies, incomplete documentation, and places where language is imprecise. They are less good at judging whether the research question is genuinely important, whether the empirical design is credible, or whether the paper is better than the prior literature. For those questions, you still need human readers (I hope!). What these tools do is take care of a lot of the mechanical work so that when you do get feedback from a colleague or a seminar, the conversation can focus on what actually matters. Let me know if it works.

Here is the Github folder with these skills.

Note: I had both Codex and Claude provide feedback on this post and help with describing the tools.