Distorted Data: Political Bias and Concerning Research Practices

Mental health professionals often rely on research to inform their practices. However, on politicized topics, the quality of that research can be affected by researchers’ political biases. These biases can interact with more general biases in the research process to compound the problem. This article reviews several examples of how political bias manifests, its psychological foundations, and what strategies practitioners can take to ensure that they are getting an accurate portrait of the scientific evidence when making decisions in their clinical work.

Keywords: Political bias, evidence-based practice, replication crisis

Mental health professionals, particularly psychologists, pride themselves on using evidence-based practices. Psychologists take courses on research methods and statistics, and as practitioners, they are expected to keep up with relevant scientific journals and learn about new developments in their field. However, evidence-based practices are only worth pursuing if the research evidence base is trustworthy. Is that generally the case?

About 15 years ago, psychology experienced a “replication crisis.” Researchers were unable to replicate many classic findings in psychology, and the field became much more aware of biases that affect how studies are designed, data are analyzed, and findings are published.¹ Since then, many steps have been taken to increase the field’s credibility, but concerns remain.

In a 2012 study, a high proportion of psychology researchers admitted to using questionable research practices, such as only reporting data for some of the outcome measures in a study or claiming that an unexpected finding had actually been predicted by the researchers from the start.² As concerning as this may be, questionable research practices are even more troubling when they interact with another problem: political bias. It is possible that the two problems could compound each other, as questionable practices are used in the service of shared political attitudes.

One type of bias has not been addressed: political bias. Most psychologists hold progressive political views, and so, to the degree that those views influence the way psychologists conduct and evaluate research, the published results tend to be skewed in a progressive direction.³ Moreover, in recent years, a variety of initiatives have asked psychologists to include political considerations in the research publication process. As a result, the research literature on politicized topics cannot represent true facts about human behavior and experiences in an unbiased way.

Consider a politically sensitive (albeit non-clinical) topic in psychology: gender differences in math performance. An early study suggested that when students are made aware of a negative stereotype that women have a lower math ability, this may cause women to score lower on a math test (the “stereotype threat” effect).⁴ This led to claims that even young girls are harmed by such stereotypes. However, a state-of-the art quantitative literature review (a meta-analysis) found that studies reporting a stereotype threat effect were more likely to get published.⁵ It seems that when researchers failed to find an effect, they either declined to share their results or journals refused to publish them. When this publication bias was accounted for, there were no significant stereotype threat effects in girls. Even so, the original 1999 study is still cited far more often than the more recent literature review.

Of course, no research study is perfect, and all studies have limitations. But political bias leads researchers to hold studies to different standards, depending on the perceived political conclusions. At times, this bias even manifests as open political pressure on authors and journals. In 2018, the journal Nature Communications published a study finding that greater ethnic diversity in researcher teams was associated with scientific papers that had more impact on their fields.⁶ The study did not raise concerns and remains well-cited. In 2020, the same journal published a second paper by the same author with the same methods, but it found that research teams containing female mentors led to fewer benefits for female mentees.⁷ This time, the methodology was harshly criticized on social media, the authors were accused of sexism (although the lead author for both papers is a woman), and under severe pressure, the paper was eventually retracted from the journal. No doubt other researchers and journal editors paid attention to this incident and will be more hesitant to publish any data that could lead to politically undesirable conclusions.

Political bias can occur at any stage of the scientific process, from the selection of questions to decisions around whether to disseminate the results to the public. However, bias is particularly powerful at the stage of data interpretation. For instance, many proponents of cultural competence in psychotherapy claim that culturally tailored psychotherapy is superior to other evidence-based psychotherapy strategies. Often, though, the studies described as supporting this conclusion never make a direct comparison. Instead, culturally tailored psychotherapy is compared to a control condition, such as being on a waitlist for psychotherapy or being provided supportive counseling.⁸ Unsurprisingly, culturally tailored therapy is found to lead to more improvement than these poor substitutes. The researchers then celebrate the effects of culturally tailored psychotherapy and demand that psychologists be trained in it, without ever demonstrating that other evidence-based approaches are inferior. Cultural tailoring may be beneficial, but given the importance of the topic, it is unprofessional to draw inferences without adequate data.

Psychologists should not be surprised by these examples of bias. Much psychological research has found evidence of people’s tendency to interpret new information through the lens of their current beliefs, with a strong motivation to find support for those beliefs.⁹ Interestingly, on partisan issues, people on the left generally do not differ from those on the right in their amount of bias.¹⁰ Indeed, in one classic study, Lord et al.¹¹ found that people who had strong views about capital punishment—in either direction—were far harsher in judging a study when it seemed to contradict their prior views.

This “confirmation bias” applies not only to research participants, but to researchers as well. If the distribution of political views was balanced across different mental health researchers, with many researchers holding views at all different points along the political spectrum, this might lead to more discussion and debate within the field. However, since most researchers in the mental health professions are on the political left, research on politically polarized issues will generally reflect a bias in that direction.

How should these problems affect mental health professionals’ understanding of “evidence-based practice?” Unfortunately, on politicized topics, where there is a strong political reason to want to find certain results, practitioners must be far more careful about accepting researchers’ pronouncements about what the evidence shows. This is most needed on politicized topics, but recent publication initiatives have encouraged journals to tie political considerations to all research. For instance, a group of scholars (including the American Psychological Association’s chief science officer) have put forth a Diversity Accountability Index that rates journals on criteria including whether the journal invites critical commentaries on any articles that have even “the potential to reinforce white supremacy.”¹²

Diversity-related criteria for publishing research articles would obviously have effects on potential authors and editors, and if a journal adheres to these criteria, readers must interpret the resulting journal articles with that in mind. More generally, the political bias in psychology research means that practitioners should be especially wary of brief summaries of studies that do not mention the details of the research methods and statistical results. Such summaries are common in practice guidelines and are particularly vulnerable to bias. Instead of blind trust, psychologists may need to use their training to locate and read the actual research studies and determine what conclusions are warranted.

The scientific method remains a powerful way to discover truth, and when used well, it reduces the biases that the human mind is prone to. Science can push back against our beliefs with clear evidence and persuade us to change our minds. It can discover which clinical practices are most effective, and it can even be used to identify effective ways of addressing concerns related to diversity and social justice. However, the scientific process is not itself immune from bias, as the replication crisis showed, and political bias may be an especially insidious type. Psychologists who wish to base their clinical practice on research evidence, as they should, must be especially attentive to political bias when reviewing that evidence.