How I learned to doubt a paper

My perspective on the science of ME/CFS and Long Covid has gone through a couple of phases since first becoming ill in October 2020.

The first phase was one of confusion, going to doctors to get tests, eventually being dismissed and experimenting with graded exercise and psychosomatic approaches. To no surprise of my fellow patients, those didn’t yield much.

Once I realized almost a year later that I might have Long Covid and ME/CFS and POTS after all, I quickly became pretty obsessed with The Science of it. I devoured every paper I could get my hands on together with other patients. I still recall pretty vividly how excited I was in late 2021 that microclots could explain it all! Now I needed to get my hands on triple anticoagulants! When that hypothesis inevitably disappointed, I moved on to viral persistence.

This phase of obsession lasted about 2.5 years. It was typified by taking most findings at face value and building elaborate theories upon these findings. It was reading Twitter, Cort Johnson, following conferences, and writing many of my own Twitter threads.

Siebe.@PatientPersists

Btw, this pathway is enhanced by lactate, which is produced during hypoxia. Sound familiar? ME/CFS has enhanced lactate levels both at rest and especially after exertion. nature.com/articles/s4159…

11:00 AM · Feb 11, 2023 · 3.02K Views

4 Replies · 16 Reposts · 106 Likes

Me, taking claims at face value and building elaborate theories.

But slowly, the cracks started to show.

I interacted with patients more scientifically literate than myself, who told me about all the shortcomings in the papers I was taking at face value. I saw findings fail to replicate. Or they implied a certain treatment would work which we already know it doesn’t because many patients have tried it. Or the findings weren’t specific to ME/CFS at all. Or they haven’t been corrected for multiple testing. And I saw Very Serious Scientists claim they had found a biomarker (e.g. cortisol) even though the existing literature already showed that it was at normal levels in other samples! (Let alone methodological issues like timing of sampling etc.)

Siebe.@PatientPersists

Alright, so this study by Klein et al. last year found serum cortisol to be significantly low in #LongCovid I was surprised at this, and recently looked into the literature. Tl;dr There are contradictory findings; cortisol doesn't seem a useful biomarker for #LongCovid 🧵

Prof. Akiko Iwasaki @VirusesImmunity

In fact, serum cortisol was the most significant individual predictor of Long COVID status in the model, and alone was a predictor wit an AUC of 0.96 (95% CI: 0.92-0.99). Notably, serum cortisol within the MY-LC study was highest in HC > CC > LC. (23/)

1:02 PM · Feb 22, 2023 · 7.55K Views

4 Replies · 6 Reposts · 47 Likes

Nowadays, I’m in the skeptical phase. I am more selective in which papers I engage with. I have a document with methodological notes that I give to Claude and ask it to evaluate any interesting-looking paper that passes by:

Evaluate objectively and rigorously. Be aware of bias, p-hacking, spin, hype, etc. But be well calibrated - when something’s good you can just say so.

Now of course, even with my extensive notes, AI has its flaws. But so do humans, and it’s a great gating mechanism to catch many weaknesses and provide context, and if the paper interests me I can check out the discussion on Science 4 ME.

But in many cases, there are severe shortcomings and the paper is a nothingburger. Occasionally, a good paper like DecodeME (a highly powered Genome-Wide Association Study) or the muscle biopsy post-PEM provocation study add something really valuable to the field.

The irony is that when patients evaluate studies from the psychosomatic paradigm, they act like great scientists. They scrutinize inclusion criteria, because many studies claim to be about ‘chronic fatigue syndrome’ but have the core feature (post-exertional malaise) as optional! They scrutinize the objectivity of outcome measures, because patients of unblinded treatments will report feeling better even when their physical activity has not increased. Like the cognitive psychologist Daniel Kahneman observed, when people defend their worldviews, it’s like they gain 15 IQ points!

Positive results are highly overrepresented in the literature due to an accumulation of biases - do not take them at face value. From De Vries et al. (2018), n = 104 antidepressant RCTs.

What really systematized all this skepticism for me was listening to the book Science Fictions by Stuart Ritchie. I think it’s a must-read for anyone seriously engaging with science. Science, we’re increasingly seeing, is not in a healthy place. Scientists are highly biased towards publishing positive results and getting citations. They have to, to keep their jobs and secure grant money. It’s extremely rare for a scientific team to publish a paper saying “we tested our hypothesis and found nothing interesting”. Naive patients getting swept up in the hype make this much worse. And the more toned down and skeptical my tweets have become, the fewer likes I get. This dynamic turns science into a hope-delivery system for instant gratification, instead of the long-term solution-delivery system it is supposed to be.

We can do better. We need to figure out how we can improve the scientific quality of our field. As I discussed in 3 levers to solve a disease, scientific quality serves as a multiplier for scientific quantity. We don’t get a lot of funding into the field. Let’s make sure that the funding we do get, counts.

I’m Siebe. I write about solving diseases — especially poorly understood ones like ME/CFS and POTS/OI, which have disabled me since 2020. Questions I focus on are:

• How can we make clinical trials cheaper and more effective?

• What makes for good science and how can we increase the quality of a field?

• What makes for effective patient advocacy?

• What does the evidence support regarding how ME/CFS works?

Find me on Twitter as @PatientPersists. Everything written with brain fog and limited spoons.