New Paper Debunks Classic Claims about Implicit Bias

This is the third post in a series on racism and implicit bias. All posts in this series appear in the new Implicit Bias and Racism section of Unsafe Science. The first two:

Implicit Bias and Racism

Recent Social Science Finds Little or No Anti-Black Discrimination

·

Jul 2

Anti-Black prejudice and discrimination was once very widespread. The obvious examples are slavery and the Jim Crow South. But discrimination was also common elsewhere, via practices such as redlining (where banks would often not provide loans to people living within areas marked off with “red lines” on a map (areas typically populated disproportionately by Black people).

The present essay is based on a paper (not mine) that recently came out disconfirming most of the classic claims of proponents of “implicit bias” — and the reason the term is in scare quotes shall soon be evident. In this post, I:

1. Summarize and document how wild and unjustified claims about “implicit bias” once dominated the academic literature and also exploded into mainstream culture.

2. Briefly summarize pre-existing evidence debunking most of those claims and provide a link to a repository of extensive sources doing so.

3. Then summarize the new paper’s results, highlighting how they disconfirmed those wild claims.

From around 2000 to 2020, “implicit bias” exploded and the term burst into the mainstream.1 Implicit bias is a topic in 150,000 academic articles!

Whole books have been written about it:

It has been turned into memes!

It has been the subject of lawsuits and mainstream media coverage!

It has even been referred to by candidates for U.S. President!

And, long before the scientific community had a chance to seriously skeptically vet common claims about implicit bias, proponents made wild claims while advocating for the research to influence law and legal practices.

In the article show above, you will find claims now known to be unjustified such as:

“The assumption that human behavior is largely under conscious control has taken a theoretical battering in recent years.”

I mean, this was in some sense true; it is just that it took another 15 years to show that the theories doing the battering were mostly made of clay.

More from the article:

69.2% of people hold implicit biases favoring White people (this is not a quote, but it is from their Table 1).

“This Article introduces implicit bias—an aspect of the new science ofunconscious mental processes that has substantial bearing on discrimination law.”

And

“The IAT is an implicit measure because it infers group-valence and group-trait associations from performances that are influenced by those associations in a manner that is not discerned by respondents.”

Uh, no. This is classic leaping to a conclusion (about “not discerned,”i.e., “unconscious”) because subsequent research (published in 2014) found that respondents “discern” (in the actual study, predicted) their IAT scores extremely well, to the tune of correlations between predictions and scores of about r=.70. Of course, the authors writing in 2006 could not have known about research published in 2014, but that is not the point. They presumed what they were measuring was “unconscious” prior to even bothering to test whether what the IAT measures is unconscious.2

“Importantly, implicit measures of bias have relatively greater predictive validity than explicit measures in situations that are socially sensitive, like racial interactions…”

This claim has been repeatedly disconfirmed but keep it in mind as you read the results of the study that is described in detail below (which examined how much implicit bias measures predicted judgments or resource allocations based on race).

“In summary, a substantial and actively accumulating body of research evidence establishes that implicit race bias is pervasive...”

Uh, no. Keep this in mind as well. This claim largely hinges on the unjustified presumption that IAT scores statistically significantly above 0 or some low threshold above 0 (e.g., .10 or .15) constitute “implicit race bias.” They don’t because IAT scores of 0 have never been empirically demonstrated to always or even usually correspond to egalitarian judgments or behaviors. Hell, IAT proponents never bothered to test this, either. Furthermore, the only publication testing this presumption (that IAT scores ~=0 constitute unbiased egalitarianism) found that scores well above 0 corresponded to egalitarianism, so that the proportion of respondents with IAT>0 (or even IAT>.15) wildly overstates the proportion of people with “implicit race bias” (and that is assuming that IAT scores are clean measures of “implicit race bias” which they are not, more on this in subsequent posts). Before getting to the recent study debunking classic claims in one fell swoop, I briefly review a whole laundry list of claims about the IAT and implicit bias that have been walked back.

Unfortunately for proponents of “implicit bias,” there has been a great walking back of many of the early claims of the advocates. Its correlations with discrimination are modest (and correlation does not mean cause anyway). When IAT scores are changed through interventions, the change in discriminatory behavior is negligible, disconfirming the notion that “implicit bias” as measured by the IAT actually causes discrimination. It is riddled with measurement artifacts, meaning it is not a clean or pure measure even of “associations” let alone racial biases. IAT scores are not particularly stable over time and no published research explains why. Whether the IAT captures meaningful differences from prejudice as measured by conventional questionnaires (a claim once routinely advanced by proponents) remains controversial, aka “not scientifically established,” with some research finding it does and others finding it does not. Go here for a repository of over 50 sources, most academic peer reviewed articles and chapters, critical of the IAT or implicit bias concept, or reporting studies justifying the claims in this paragraph and disconfirming common claims about the IAT or implicit bias.

And, most recently, another new paper has come out finding that much of the “bias” in the IAT can be explained by the metaphorical use of “black” and “white” to represent good and evil culture-wide. But that, too, will be a post for another day.

Anthony Greenwald, the creator of the IAT, and I, were presenters at a 2017 NSF conference on the controversies surrounding it and implicit bias more generally. Chapter versions of those presentations can be found here:

At that conference, Greenwald presented this as the working definition of “implicit bias” used by scholars up to that time (and, I’d argue, it is what many still assume it means).

Unfortunately, and despite the earnestness with which he presented it, it is nonsense. No, really. That it is literal nonsense should be detectable by a moderately intelligent 9th grader. This slide I use when giving talks about implicit bias and the IAT unpacks how much nonsense is packed into this nonsense definition. In the slide I simply crossed out every component of his definition that is either logically incoherent or empirically unjustified:

Reaction times, which are what the IAT measures, are not discrimination. A 9th grader should be able to realize that, so we have:

“Introspectively unidentified (or inaccurately identified) effects of past experience that mediate discriminatory behavior.”

Even without statistical training, anyone who realizes that that “mediation” means “A causes C because A first causes B and B causes C” will also realize that mediation requires at least three variables. The IAT is one variable. Its ability to mediate stuff could be tested but no single variable can possibly be defined as mediating anything.3 Now we have:

Introspectively unidentified (or inaccurately identified) effects of past experience that mediate discriminatory behavior.”

IAT scores were long presumed to be unconscious without evidence demonstrating unconsciousness — and when whether they were or not was actually empirically tested, bingo! — people knew what their IAT scores would be. Although a reasonably intelligent 9th grader could not be expected to know this peer reviewed article, here is what said 9th grader could know — just because some human response is fast does not mean it is “unconscious.” Baseball and tennis players hit balls hammered at them with response times measured in milliseconds. When a car cuts you off on the highway, you hit the brake all-but-instantaneously. None of those responses are “unconscious” — so, absence evidence of unconsciousness (which has never appeared), no one should have ever assumed the responses on the IAT were unconscious. Now we have:

Introspectively unidentified (or inaccurately identified) effects of past experience that mediate discriminatory behavior.”

After removing all the unjustified nonsense all that is left is “effects of past experience.” Well, isn’t everything?

If Greenwald was right, and that’s what social scientists thought they were doing when they administered the IAT, they were deeply confused. And the social science literature is filled with stuff that presumes all or most of this nonsense, which is the type of thing that motivated this Unsafe Science entry:

~75% of Psychology Claims are False

·

November 1, 2024

In this essay, I explain why, if the only thing you know is that something is published in a psychology peer reviewed journal, or book or book chapter, or presented at a conference, you should simply disbelieve it, pending confirmation by multiple independent researchers in the future.

Ok, you now have sufficient context to understand the falsification power of the paper that is at the center of this post.

Unfortunately, the paper is behind a paywall, though probably available through many university libraries. Fortunately, however, it is available in full at the repository of articles critical of implicit bias and the IAT I created some time ago.

This paper was an adversarial collaboration between those who believed in pervasive racist biases and those who did not. Each author also presented their specific predictions for the study prior to conducting it (available in the supplement), also making it effectively a registered report.4 Adversarial collaborations’ great strength is that authors with competing theories, hypotheses, and perspectives constitute a skeptical check on one another, thereby improving methodological rigor and increasing the likelihood that conclusions and interpretations have close fidelity to the actual findings. Our paper on adversarial collaborations argued this occurs through the processes shown here (go here for the Substack version):

In our paper, we argued that the combination of adversarial collaboration and registered report constitutes as close as you can get to a gold standard in rigor and credibility in social science. So the Axt et al paper lined up its quality ducks in advance very nicely.

1. How much racial discrimination would they find?

2. How much would racial discrimination be predicted by:

   1. Explicit conventional questionnaire measures of prejudice?

   2. Implicit bias measures?

3. Which would more powerfully predict racial discrimination, questionnaire measures of prejudice or implicit bias measures, in models entering them both simultaneously?

Sample size: 2114. This exceeds the sample size in the vast majority of social psychology studies.

Discrimination measures: 1. the Ultimatum Game (involves allocation of money to other players); a Trust Game (also allocation of money to other players), a Judgment Bias Task (a hiring decision), and a Resume Evaluation Task (also a hiring decision). Here is what they wrote about how they selected these particular measures:

In particular, we sought measures that (a) were deployable online, (b) elicited overall evidence of anti-Black discrimination in prior work, (c) had been linked with indirect measures in prior work, and (d) provide an individual-level index of participants’ relative tendencies to discriminate.

Implicit bias measures: The conventional IAT (explained in an Appendix at the end of this post), the single category IAT, an Evaluative Priming Task and the Affect Misattribution Procedure (sorry, it would take too long to explain the other three not in the Appendix, but they are easy enough to find via Google Scholar if you are so interested. They are also described in detail in Axt et al’s supplement which is also available in my online repository of articles critical of the IAT and implicit bias). Here is what they wrote about how they selected these:

When selecting indirect measures, we considered instruments’ widespread adoption, psychometric properties, convergence with other indirect measures, and prior evidence of predictive validity. We also sought methodological diversity to minimize shared method variance.

Explicit prejudice questionnaire measures. There were five single item direct preference measures, described as follows:

We included a set of five direct preference measures (DPMs) capturing self-reported preferences and relative feelings about Black versus White people (Axt, 2018). Item 1 was a multiple choice question measuring relative preference for Black compared with White people (“Which statement best describes you?” with seven response options ranging from I strongly prefer Black people to White people to I strongly prefer White people to Black people). Items 2 and 3 were multiple choice questions measuring liking of White and Black people (“On average, how much do you like White/Black people?” with seven response options ranging from strongly like to strongly dislike). Items 4 and 5 were sliders measuring positivity of feelings of White and Black people (“How negative or positive are your feelings toward White/Black people?” With labels of extremely positive and extremely negative at the slider endpoints and neutral at the midpoint).

These were combined to form an overall Direct Preference Measure (DPM).

They also included a six item measure of racial stereotyping from the American National Election Study, a previously published 10-item Prejudice Index, and the Symbolic Racism Scale, one of the most commonly used measures of prejudice out there.

So, there were also four questionnaire measures of prejudice: DPM, ANES/stereotyping, Prejudice Index, and Symbolic Racism.

1. There was no evidence of anti-Black discrimination on any of their four measures. Behavior was unbiased on the Ultimatum game and it was pro-Black (or anti-White) on each of the other three discrimination measures.

2. Correlations of the four implicit measures with discrimination hovered barely above 0. From the paper (Figure 1):

These results do not support claims about the power of implicit biases so popular among the early advocates and in mainstream culture.

3. The correlations of questionnaire measures of prejudice with discrimination ranged from pretty low to pretty substantial and were consistently higher than those for the implicit measures:

So its not like “prejudice is dead” — its just that, contra 1990s and 2000s claims about prejudice going underground, it was entirely above ground, i.e., readily measurable by conventional questionnaire techniques. The lead author, Jordan Axt, also has this excellent paper on that general topic (which I highly recommend to anyone wishing to identify strong measures of prejudice):

4. They (the more recent paper) also performed structural equation modeling5 to assess the overall relationship of implicit and explicit attitudes to discriminatory behavior. This technique removes measurement error from the estimates, which has the potential to reveal stronger relationships than does assessment of relationships (e.g., via correlations or regression) among measured variables. The particular model they assessed tested for effects of implicit and explicit attitudes simultaneously, which means the obtained coefficients for each constitute the marginal effect of each, after accounting for the other. They referred to this as incremental validity, because the coefficient for implicit attitudes constitutes its predictive validity for discrimination over and above explicit attitudes; and the coefficient for explicit attitudes constitutes its predictive validity for discrimination over and above implicit attitudes.

   Because of the large sample, even very small coefficients could be “statistically significant” (which you can see in the two tables of correlations shown above). Therefore, the authors agreed on a threshold of .15 (analogous to a standardized beta from regression) for considering a relationship nontrivial.

   So what did their structural equation modeling analyses find? The key findings were these: The incremental validity for the explicit measures was one of the largest effects ever obtained in social psychology: .67 (the average effect in social psych is about r=.20, effects are rarely above r=.40; in other words, the explicit measure effect was very large). The incremental validity for the implicit measures was .16, i.e., right on the border of their agreed-upon threshold for triviality. The confidence interval for this coefficient ranged from .05 to .27, which means it is possible that (by their standards) the true relationship is trivial; and its possible that the true relationship is nontrivial. So whether the result was trivial or not is ambiguous (scientific uncertainty lives!). Regardless, these results disconfirmed classic claims about the power of implicit bias relative to explict prejudice.

Here is what they wrote:

At least among these measures of discrimination, participants who displayed higher levels of bias on indirect measures were only less pro-Black or race-neutral in their behavior, rather than engaging in outright anti-Black discrimination (see also Axt et al., 2016; Blanton et al., 2009). Only a relatively small subset of participants with extreme pro-White scores on indirect measures (7.61% of the sample) engaged in statistically significant discrimination against Black relative to White targets, and to only a small degree.

This study disconfirmed nearly all the wild claims about “implicit bias” that led off this post — including those appearing in the peer reviewed and legal literatures and in popular culture.

• The ability of implicit measures to predict discriminatory behavior was right on the threshold of trivial.

• Most of what there was to predict was egalitarian or pro-Black discrimination. However, because there was some anti-Black discrimination, it is fair to conclude that the implicit measures predicted that (albeit weakly). However, most of what implicit measures predicted was not anti-Black discrimination. Instead, they mostly predicted variability in discrimination favoring Black people to unbiased behavior because that was mostly what was there to be predicted.

• And despite using four measures of discrimination that had captured anti-Black discrimination in past research, they found no net anti-Black discrimination. Overall responses were egalitarian on one measure and favored Black people on three measures. A very small proportion of the sample — under 8% — actually engaged in anti-Black discrimination. This is the same as saying 92% did not (most of whom engaged in pro-Black discrimination).

In a 2006 issue of the California Law Review Kang and Banaji presented work on implicit bias as justifying nearly endless affirmative action:

They stated:

Most fundamental is the pervasive, replicable, and sometimes large effects of implicit bias in the here and now.Implicit biases are not merely an academic concern, although their discovery has shaped new theories of mental processes. Implicit bias has consequences in the daily activities of our lives. Indeed, on socially sensitive matters such as discrimination, implicit bias scores have greater predictive validity than explicit self-reports. The assumption is that individuals are not necessarily withholding their “true” attitudes and beliefs but rather that they are unable to know the contents of their mind.

And, most amusingly:

A model that supposes that discrimination takes place explicitly, through a rational cost-benefit analysis or other expression of explicitly held views has become woefully out-of-date.

And here is the call for affirmative action on the basis of the very dubious IAT and the even more dubious presumption that scores of 0 on the IAT constitute egalitarianism:

“Fair measures that are race or gender-conscious will become presumptively unnecessary when the nation’s implicit bias against those social categories goes to zero or its negligible behavioral equivalent.”

Academic biases in promoting “social justice” notions — such as massive anti-Black discrimination and the power and pervasiveness of implicit biases — are legion. Go here for an entry on the quasi-religious and highly unevidenced nature of claims about “systemic” racism (now scaled up into a more sophisticated version published in a peer reviewed academic journal). Go here for a series of articles on similar academic biases in promoting research on sex discrimination in academic hiring (which finds far more evidence of hiring biases against men than women). Go here for a post on academic delusions about what the research sex bias in hiring more generally finds.

There is good news, here, though, at least for reasonable people. The evidence in the Axt et al study, and indeed in research on race and sex discrimination more generally finds, not that such discrimination is dead, but that it is the unusual exception rather than the general rule. That is good news for everyone except academics who make their careers promoting such discrimination as powerful and pervasive.

Here is the basic setup for an IAT. It is a double categorization, reaction time task, done on a computer. Someone taking an IAT might face something like this Obviously, what is shown below is a race/attitude IAT, but IATs can be constructed for all sorts of groups (sex, ethnicity, religion, even for individual people) and they can be constructed to assess beliefs (smart/dumb, strong/weak, etc.) and not just attitudes.

Participants will see something like this on a computer screen:

IAT 1:

The top row are the categories. The bottom four words and names are the target words. In a real IAT they appear one at a time, not four at once as shown here. The participant’s job is, using computer keys, to indicate where the target word belongs. If it is EITHER Black or Pleasant, that’s where it goes. Rainbows are pleasant so they go to Black/Pleasant. Jamal is a Black name so Jamal also goes to Black/Pleasant. Vomit is unpleasant so it goes to White/Pleasant. Etc.

BUT the IAT is not done yet. After doing this (or maybe before, the order is not particularly important), they have to do it again:

IAT 2:

Rainbows are pleasant so now go to White/Pleasant. Jamal is a Black name, so he goes to Black/Unpleasant. (In a real IAT, Black/Unpleasant and White/Pleasant will again appear at the top of the computer screen, not the bottom, as shown here).

The main outcome of an IAT is a speed difference. The technique is programmed to measure the time it takes for participants to categorize each target word in IAT 1 and IAT 2. The IAT Score is obtained by taking IAT1 minus IAT2. (There is also a whole statistical standardization procedure that is beyond the scope of this appendix).

The idea is this: If people find it harder to categorize words into Black/Pleasant and White/Unpleasant than into Black/Unpleasant and White/Pleasant it means there is a stronger psychological association between Black and Unpleasant, and White and Pleasant, than between Black and Pleasant and White and Unpleasant.

This was meant to and once widely believed to measure “unconscious racism.” And, unless the difference score is near 0 (reaction times for IAT1 ~=IAT2), it is interpreted as reflecting “implicit bias.” Why its not interpreted as reflecting “a difference in reaction times” rather than “implicit bias” was never scientifically clear to me, given that differences in reaction times are no sort of bias whatsoever. Like, if it takes me longer to run a mile when its 90 degrees out than when its 55 degrees out, do I have an implicit bias against heat? If it takes me longer to cook a steak than an egg, do I have an implicit bias against eggs? Obviously, those are rhetorical questions and no one (in their right mind) would say “yes” to either question. Ok, then, so why is taking longer to do one double categorization task than another a “bias”? Sorry, that was another rhetorical question; its not a bias. It is a “difference in speed.” Of course, I do understand why it was interpreted this way — it made those advocating for this type of research seem very serious and scientific, and as if they were addressing something psychologically, socially, and politically important — “unconscious racism.”

IAT scores were eventually shown to be very much not unconscious, so those who embraced it had to do some fast conceptual shuffling when they described it. So now that “unconscious” is known to be unjustified, you will find all sorts of other descriptions in the academic literature. IATs indirectly measure racial (or other) attitudes. IATs measure automatic attitudes, etc. What, exactly, these terms mean is a bit of a conceptual mess (see, e.g., here or here), but it sure feels good to talk about automatic attitudes and indirect measurements, doesn’t it? Very high in scientificiness, a close relation of truthiness, but more prestigious.

Before commenting, please review my commenting guidelines. They will prevent your comments from being deleted. Here are the core ideas:

1. Don’t attack or insult the author or other commenters.

2. Stay relevant to the post.

3. Keep it short.

4. Do not dominate a comment thread.

5. Do not mindread, its a loser’s game.

6. Don’t tell me how to run Unsafe Science or what to post. (Guest essays are welcome and inquiries about doing one should be submitted by email).

Footnotes