Greetings from The Metascience Observatory! A lot has happened in the last three months.
The Metascience Observatory is pioneering a new form of massive AI-powered literature review we call the “Bird’s Eye Review”. The philosophy behind our approach is described here. A write-up about findings from the Long COVID review will be posted on Substack soon. If you see value in this sort of work, please consider donating - we are reliant on donations to be able to continue this work.
Initially we focused on importing “direct” or “very close” replications, but recently we have started importing “close extensions” and “conceptual replications”. We found that “replications” in genetics are almost always “close extensions”, for instance looking for a genetic correlation in a different population. A new dropdown on the replications database page allows users to filter the database by replication type. You can read about how we define the different types of replication here.
Since January we’ve had pages where you can view replication rate by discipline/subdiscipline and by journal. Across fields where we have a lot of data (psychology, neuroscience, economics, a few others), we’re finding replication rates tend to be in a narrow range between 40% - 60%. We also found that reproducibility has not changed much over the course of the last fifty years, which is either reassuring or disturbing depending on your point of view.
Replication initiatives are concerted efforts where a group of researchers try to replicate a number of studies in a particular field or journal. We’ve documented 17 initiatives and have incorporated data from them into our dataset.
Psychology
2014 Many Labs 1 RR=77% (10/13)
2014 Social Psychology Special Issue on Replication Reports RR=32% (14/44)
2015 Reproducibility Project: Psychology RR=36% (36/100)
2018 Experimental Philosophy – Reproducibility Project RR=70% (28/40)
2018 Many Labs 2 RR=46% (11/24)
2019 Soto – Life Outcomes of Personality Replication Project RR=87% (68/78)
2022 Motoki and Iseki – Sensory Marketing Replication RR=20% (2/10)
2023 Boyce et al. – Student Replication Projects RR=49% (86/176)|
(ongoing) Clearer Thinking’s Transparent Replications Project RR=83% (10/12)
Social Science
2018 Camerer et al. – Nature/Science Social Science Replication Project RR=62% (13/21)
2025 Examining Replicability of Online Experiments RR=54% (14/26)
2026 DARPA SCORE Social Sciences Replication Project RR=55% (151/274)
Biomedical
2021 Reproducibility Project: Cancer Biology RR=57% (107/188)
2025 Brazilian Reproducibility Initiative RR=31% (30/95)
Economics
2016 Experimental Economics Replication Project RR=61% (11/18)
(ongoing) International Initiative for Impact Evaluation Replication Papers RR=90% (9/10)
Sports & Exercise Science
2025 Estimating the Replicability of Sports and Exercise Science Research RR=28% (7/25)
pdf4llm - most multimodal AIs (like ChatGPT, Opus) read PDFs by converting them to images. While this is great for understanding images and figures, it can lead to slight errors reading text, especially small text like numbers in tables. In an internal study, we found slightly better LLM extraction performance after converting PDFs to markdown. There are many tools for converting pdf to markdown including docling, grobid, tesseract, and pymupdf, but they all have strengths and weaknesses. Docling is great for tables, but fails with scanned PDFs and “weird” PDFs. Grobid excels at reference extraction but isn’t good at tables. Tesseract is good at scanned PDFs and tables that are images. Pymupdf is a good fallback for “weird pdfs”. The Metascience Observatory’s pdf4llm package orchestrates all four different tools to get the highest performance. In the end, pdf4llm converts a pdf into abstract.md, body.md, references.md and references.json.
fetchpdf - this is a simple script that calls six or seven different APIs to try to download a pdf from a DOI or PMID.
Inspired by the work of James Heathers, Elizabeth Bik, Sholto David, and Markus Englund, the Metascience Observatory is developing a specialized AI agent for forensic metascience. To learn more and get involved, please reach out.
Op-ed by Eugenie Reich : “A call to end the ‘impact on conclusions’ test for retraction”
Cell Press’s mega journal Heliyon retracted 392 papers in 2025. – The retractions come an internal audit that Heliyon started in response to pressure from metascience activists.
The Lancet refuses to retract or put a warning on a 1989 Letter to the Editor – the letter discusses and promotes fraudulent work on Crohn’s disease.
arXiv preprint - fake/bad reference citations started appearing in 2021 (the year ChatGPT came out). Between 2021-2025, they comprise 2-6% of references in high-performance computing conferences.
Elizabeth Bik reports on a new way she is catching fraud - finding unexpected peaks in X-ray diffraction spectra.
Nature Immunology has retracted two papers for improper image duplication - one of those papers had over 1,000 citations.
Indian researcher calls on the government to incorporate retraction data into university rankings.
Elizabeth Bik published a great overview and retrospective on image fraud. - Fraudsters are now using generative AI, so fraud detection needs to evolve as well. The Metascience Observatory’s forensic metascience agent will help on this front.
Cabell’s list of predatory journals grows to 20,000 journals. - The Metascience Observatory is building a free version of Cabell’s journal dashboard at
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.