In Which The Rube Goldberg Machine Behind Retractions Delivers A Limp Payload A Year Late

It’s been a while, because I’m building the MEP.

I don’t have a writing schedule that I keep to, so writing this sort of thing loses out — there hasn’t been much time for dyspepsia, billingsgate, and opprobrium.

But something is on my mind, so here we are.

In March 2024, the BMJ Group put out a press release:

In May 2024, I noticed the study was really bad, and said so. It was full of unlikely features, contradictory data, and generalized silliness.

Then two legends, Vahid Malbouby and Eric Trexler, write to me — they have their own additional reasons to distrust this paper. We combine all the most alarming reasons into a single correspondence and send it to the journal.

They receive it in June, it is accepted in October.

November/December 2024, the authors write a nonsensical rejoinder, and it is published.

January 2025. Eric notices that the rejoinder is published, but our correspondence isn’t. The rejoinder is rejoindering to a paper that doesn’t exist yet — so they can’t cite us. We politely remind the journal that as time is linear, this represents some kind of mistake.

February comes, and they publish our letter. They stealth-correct the authors rejoinder so it doesn’t look like they were hallucinating.

And, the story ends today, as it started, with a press release:

559 days since publication.

504 days since my first criticisms were public.

464 days since the journal has had a copy of the combined criticism.

343 days since they agreed to publish it.

$2652, the amount the journal initially (and erroneously) asked for to publish the letter.

12, the amount of emails it took to resolve the payment waiver.

~100, the amount of emails sent or received in pursuit of this nonsense.

0, the amount of emails from the journal to tell me this retraction was happening.

Obviously “This retraction reflects our strategic and proactive approach to investigating concerns raised about the content we publish.” is not going to have a lot of resonance with me. Not even through the most twisted process could the above be described as proactive, considering how much external action was necessary.

But let’s not get side-tracked before we start — I want to get to this bit:

“While we deal with allegations as swiftly as possible, it’s very important that due process is followed.”

If there’s a genre trope of writing about research integrity that we could lean into here, and this particular dyspeptic screed was written by someone else, this would be the point where I point out that this journal is bad at research integrity.

Here’s the thing: I’m not sure we’re getting the whole story if we stop there.

Because it’s not really an explanation, it’s an outcome. Several things can simultaneously be true in the explanation of that outcome:

• The lawyers can get involved. Both external and internal. Lawyers are slow, risk-averse and expensive. I have heard stories about RI teams wanting to act and being hamstrung by lawyers. Bear in mind, lawyers are non-scientists paid to be over-protective and paranoid in the service of their employer’s interests.

• Everything a RI team does needs to be pushed through the internal culture of a journal. I have heard stories about RI teams wanting to act and being hamstrung by editors.

• Multiple parties mean lag. Who’s involved? The author(s), the institution(s) of employment, whoever is handling the data concerns, staff at the journal, the RI team, the lawyers, and staff from the publisher side (probably in multiple roles). Just typing all that out gives me a cold sweat. I can see months being lost because some dim bulb missed an email that came in during a golf lesson.

• Authors stonewalling deserves its own mention here — run out the time on initial requests, wait for an angry follow-up, then send half-hearted nonsense, and basically string the entire process out as long as possible in the hope that everyone gets bored. This can work, not because the case necessarily goes away, but because people change jobs. What if the editor is on a two year contract? You can run out the clock on that — then you get a new editor, who inherits a barely comprehensible mess which needs to be begun from scratch. In these PubPeer cases where papers are retracted seven, eight years after being flagged for something serious, I wonder how many times the person in charge of that portfolio has moved or been replaced.

  
  and then there’s the point which I think a lot of people are rather unpleasantly implying when we say ‘journals are bad at research integrity’, which is…
  

• The investigators are poorly organized, ignorant, under-trained or disinterested. And, I mean, maybe? I’m not convinced that’s the case.

Speaking of which!

Three things come to mind.

(1) get better at investigating problems

There’s a weird two-speed investigatory culture in research integrity. Generally:

• Forensic metascientists / scientometricians / sleuths / whatever are generally statistically competent scientists, who learn more about investigative methods over time, are interested in the topic outside of their professional work and learn passively (because we read each other’s crap!), and work proactively.

• Research integrity staff generally hire statisticians, aren’t scientists, have a career in publishing, work passively (they respond to problems rather than pursue them), and would rather put their hand in a blender than voluntarily participate in learning more about statistical arcana.

This is relevant because: none of what you need to know to be able to investigate a computational research problem fairly, correctly and quickly is spooky knowledge from the great beyond.

I firmly believe these skills can be learned, can be taught, and can be retained within an organization. Or I wouldn’t be involved in starting a goddamn research project to do exactly that, would I?

Sadly, there is very little two-way traffic between these two groups of people. I’m not really in a position to improve this … yet.

However, if there is one thing that we can be sure of, it is that the one place in the whole palaver of a full retraction cycle where you can save time is the investigation. What people do with the results of that investigation, different story.

(2) for the love of God, please use the Expression of Concern / Editor’s Note function

COPE are the big non-profit who try to define best ethical best practices in scholarly publishing. COPE’s main job, as far as I can tell, is issuing guidelines that its members — mostly large and mid-sized academic publishers — routinely ignore.

COPE have previously been described by many people as a ‘toothless tiger’. I would classify them more as a legless tiger, a kind of striped sausage. They have all the statutory authority of a child dressed as a cop for Halloween and knuckles made of silken tofu.

And even COPE recommend that you flag potential or ongoing research integrity cases with some kind of publishable note.

The COPE Retraction Guidelines describe when journals could use expressions of concern. For example, editors should consider an expression of concern if:

• they receive inconclusive evidence of research or publication misconduct by the authors

• there is evidence that the findings are unreliable but the authors’ institution will not investigate the case

• they believe that an investigation into alleged misconduct related to the publication either has not been, or would not be, fair and impartial or conclusive

• an investigation is underway but a judgement will not be available for a considerable time

LINK

I do not know why journals persistently refuse to use these notices. I don’t have any strategic or operational insight into why this option is so routinely ignored. Someone let me know if you know.

But it would very much help. So, so often in an investigation it should be incredibly clear very early on that something will eventually happen to the paper. At that point in time, it can be flagged — and yes, flagged without defaming the authors or causing a legal incident.

(3) go to whoever sets your budgets, and set their shoes on fire

Research integrity work is underfunded everywhere. It is my present impression that this includes ‘at publishers themselves’.

The abjectly pathetic results at work in this particular case are almost certainly affected by the fact that “good, fast, cheap: pick two” operates at all levels of all organizations, and what we’d really like to push back against the growing torrent of research horseshit is a system that is good and fast.

I think before this changes, and publishers start making Serious Investments in their RI processes, there will have to be An Incident.

Maybe several Incidents.

I do not know what the Incident will be, but likely some kind of scandal where there is a direct and seriously problematic international scandal, probably one involving a lot of lost money and dead people, which is directly and unambiguously provoked by a research quality issue.

Likewise, something else I’ve been saying for years — there’s no quality differentiation from publishers that matters much, only from journals. Publishers are ‘real’ or they’re somewhere on the long tail of ‘less than good’. There aren’t any good ones, only ‘good enough’ ones.

The first publisher to actually convince people that they’re actually good at managing research integrity problems, which are just going to continue to multiply over the next few years and become even more outrageous, will occupy a very interesting brand space.

The case for a serious internal investment in research infrastructure needs to be made, but there likely isn’t anyone outside the senior leadership of an academic publisher who can make it.

So, if you’re out there, call me. I’ll still be here, trying to make what we all give the world less awful.

Stay hard.