I enjoyed making a presentation titled “Questioning Some Metascience Assumptions” to the Association for Interdisciplinary Meta-Research and Open Science (AIMOS) at Victoria University of Wellington, New Zealand on 13th August. There’s a recording of my presentation below and a copy of my slides here. (The recording appears to cut out towards the end, but bear with it and it will get back on track.)
Many thanks to Jan Feld for the invitation and to the audience for their interesting questions.
Jenni Adams and colleagues published an article that “considers openness, transparency and rigour across different subject areas and research types.” As they explained:
Applying a unitary model of openness that aligns primarily with quantitative and positivist research paradigms risks perpetuating the devaluation of AHSS fields [arts, humanities and social sciences] (Leonelli 2023; Ross-Hellauer 2022), homogenising research approaches (Bazzoli 2022; Field 2025; Guzzo et al 2022), and sustaining AHSS researchers’ disengagement from open research. In this context, it is crucial for researchers across a range of methodologies and epistemologies to be given the space to work through what openness can meaningfully look like in their research.
Relatedly, it was good to see Grant et al. (2026) reflecting on the concern about the applicability of open science practices in their recent update to the Transparency and Openness Promotion Guidelines (TOP 2015). Specifically, they acknowledged that TOP2015
was developed primarily (though not exclusively) by social and behavioral scientists concerned about misuse of null-hypothesis significance testing…, selective reporting, and computational irreproducibility in published experimental research…. As a result, the full set of TOP standards is most directly applicable to quantitative studies and other designs that test questions or hypotheses about potentially generalizable knowledge. Some standards may be less applicable or less important for other epistemologies, study designs, or phenomena of inquiry. (p. 6)
Returning to Adams et al.’s (2026) study, there’s lots of interesting findings here that highlight the heterogeneity of perspectives on openness, transparency, and rigor. However, I was interested in participants’ perceived practical constraints of transparency. In particular, the finding that caught my eye is reported in this single sentence:
Some participants also expressed concern that sharing too much information may obscure rather than clarify.
The concern about transparency obscuring rather than clarifying has been discussed previously. For example, Elliot (2020) argued that “transparency can also be dangerous. It has the potential to confuse people by burying them under a flood of information that they cannot easily digest” (p. 349).
This issue also relates to my own concerns about “mindless transparency.” It’s not possible to be transparent about everything in a research study. We need to make a selection of information on the grounds that we believe it to be relevant to our research claims. Nosek and Bar-Anan (2012) made the same point:
Published articles do not report everything that was done or found. They cannot do so. Reports of a study’s methodology reflect the researchers’ best understanding of what is crucial in the design. However, this almost always reflects a qualitative assessment of the key factors of the sample, setting, procedures, measures, and context. In other words, the reported methodology describes what the researcher thinks is important, not necessarily what is actually important. (p. 225)
Consequently, in my view, it’s important to distinguish between relevant and irrelevant transparency because it’s only useful to ensure transparency about relevant information, not irrelevant information (Rubin, 2020, p. 383; see also Fiedler and Schwarz, 2016, p. 46; Leonelli, 2023; Popper, 1962, p. 230). As Popper (1962, p. 230) put it, the principle of reporting “the truth, the whole truth, and nothing but the truth” only pertains to the relevant truth. Indeed, mindless transparency that fills research reports with irrelevant information may impede the efficient communication and understanding of research claims. Hence, mindless transparency can be detrimental to effective open science, and one of the most important open science practices is to think carefully about what we’re being transparent about and why.
Of course, researchers may be biased when determining what information is “relevant” and “irrelevant,” and different epistemic communities may have different norms about what they consider to be “relevant” (e.g., depending on different philosophies of science and statistics). However, where there’s disagreement, people can request transparency about the information they think is relevant, and if that transparency is not forthcoming, they can downgrade their evaluation of research claims. Hence, the potential for bias and disagreement about relevance does not warrant indiscriminate transparency that has the potential “to obscure rather than clarify,” to quote Adams et al.’s (2026) participants.
Adams, J., He, S., & Zagrodzka, Z. B. (2026). Leading cross-disciplinary conversations around open, transparent and rigorous research: Findings from a series of focus groups. Scottish Journal of Open Research. https://doi.org/10.36399/kgdnt160
Mark Ramos published an article in the statistics magazine Significance in which he comments on the recent Systematizing Confidence in Open Research and Evidence (SCORE) project. News reports of the SCORE project’s 55% replication rate (Tyner et al., 2026) took a “glass half empty” view, claiming that:
Only about half of social science results can be replicated, finds new study
Across the social sciences, half of research doesn’t replicate
Half of social-science studies fail replication test in years-long project
I’ve recently queried this “glass half empty” interpretation here, and Ilka Gleibs has provided a “glass half full” perspective here. Ramos has a similarly optimistic view. He concludes, that
reasonably speaking, having a 50% replication rate in social and behavioural studies is much better than it has been made to sound. The results from the SCORE project should be considered a reassuring sign of good science prevailing.
A summary of Ramos’ (2026) argument is as follows:
A balanced perspective on positive publication bias recognises that although publishers’ preferential treatment of statistically significant studies when selecting among studies of otherwise comparable quality inflates the false positive rate in the published literature beyond nominal per study levels, it also increases the rate of true discoveries. Moreover, the nature of statistical significance testing is such that rejections of the null hypothesis carry more actionable information than failure to do so. As such, evaluating this phenomenon requires more nuance than focusing solely on its effects on false positive rates. Under reasonable hypothetical scenarios, such as one where selecting a null hypothesis to study that was actually false is rare, positive publication bias does not severely inflate false positive rates in published literature as long as all researchers are acting in good faith – that is, their nominally reported significance levels are consistent with all design and analysis decisions made in their study.
Ramos posted the following cartoon on BlueSky, which I think works well to illustrate his position:
See also Ramos’ related work in this area:
Ramos, M. L. F. (2026). Balanced examination of positive publication bias impact. Accountability in Research, 33(4). https://doi.org/10.1080/08989621.2025.2538066
For your edification, here are the top 10 posts from the Critical Metascience Substack in descending order of number of views:
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.