Re-approaching Peer Review in the Age of LLMs

Just in the past week, the following post crossed my mastodon timeline:

I have spent several hours writing feedback on this paper, and now I am looking at the references and … none of them exist. Next time I am starting with the references 😠 @mark@mastodon.ocert.at

and a friend who’s a journal editor sent me this from bsky:

The linked study cites an article which lists me as an author, but which does not actually exist.

I complained to the journal editors.

Over a year later, the study is still there, no retractions or corrections. pubmed.ncbi.nlm.nih.gov/39614949/

December 20, 2025 at 2:42 PM @thewonkologist.bsky.social

… much like the first person quoted, I tend to get to references last. As I’ve matured, I had already become aware of the long-standing problem of mis-citation where an author cites a real publication but either misuses or misunderstands it (sometimes it’s obvious which, I try to be generous when it’s not). So I do actually check references. At this point, there are some classics I can recognize. But if I don’t, I will actually do a quick vibe check skim, maybe something deeper when the claim is either really important to what I’m reviewing or I find it surprising.

Inverting the Process

But clearly, the era of LLMs calls for a new approach, checking references first. Even a “grounded” LLM isn’t guaranteed not to spit out a citation that doesn’t exist. After checking for existence, it may be even more important now to check if a work is being cited appropriately. The presence of a widely-cited article in a bibliography might just mean it’s significant in the corpus but not used correctly here.

For my own part, the presence of a single reference whose existence I cannot confirm will be enough for me to stop review.¹ I am not going to put in work for same people who could not be bothered to do their own work.

You are already not owed publication for heartfelt work, but you are owed respect and engagement. That’s what the peer review process is for. You are not owed my review time for something you couldn’t bother to put time into.

What Next?

What should the editors do? The Library Loon proposes automatic rejections (and career consequences) and I’d agree that’s the option most in line with “you are owed the degree of work you put into it.” I’d also note that fabricating references, when done intentionally by humans, was already considered research misconduct. I don’t think being too lazy to check the a tool’s work makes it not misconduct, just embarassing too.

The most generous thing I could come up with would be requiring the provision of links to prove that the cited works exist (not access, but at least existence) along with text excerpts of the relevant paragraphs with page numbers so that editors or reviewers can check the work (and do a quick visual comparison with articles/works they can access). But editors and reviewers are already overstretched trying to engage with people who actually did their own work. In a world where we are all already so overstretched, why should they have to put in all this extra time to check someone’s work when that person couldn’t be bothered to check it before submitting?

As peer reviewers, we can’t stop people from trying to fill up journals with slop like so many landfills. But we can treat it the same way we would an author who made up all their citations in 2010: rejection, end of conversation. And in changing how we approach reviews, bibliography-first, we can stop them from wasting (much of) our time.