Do Partisans Believe What They Say?

When Americans are asked factual political questions, their answers divide along partisan lines. Is climate change happening, and is it mainly caused by humans? In our sample, 98% of Democrats said yes, but only 45% of Republicans agreed. Has illegal immigration increased during Biden’s presidency compared with Trump’s? Again, 91% of Republicans said yes, while only 25% of Democrats agreed. These are not disagreements about values or the best policy programs, which are not straightfowardly factual matters. Rather, they concern plainly factual issues. Yet the answers seem to track party affiliation instead of the evidence available.

So, do partisan live in radically different realities? Not necessarily. It may be that survey response often do not reflect the statement of a belief. Rather, factual disagreement can sometimes reflect partisan cheerleading. The two are difficult to tell apart simply by looking at the responses themselves.

In this post, I report data from a new, as yet unpublished study that I conducted with Olaf Borghi and Irena Arslanova, designed to test whether partisans are genuinely divided on factual political issues. Our central finding is that much of the partisan “disagreement” over facts may not be sincere disagreement at all. But the results were stranger than expected. When we paid people to answer accurately, only one side changed its answers. And paying for accuracy did not always increase accuracy. Before I turn to the results, let me provide a bit more context.

Explanations for Partisan Gaps

Why do people disagree about the facts? There are two broad possibilities. The first is that the gaps reflect sincere differences in belief. That disagreement might arise through motivated reasoning, where people process information in ways that protect their political identity and end up sincerely believing what flatters their side. Or it might arise without any motivated cognition at all. Rather, it’s that Democrats and Republicans consume different media, hear different claims, and follow different elites, so their beliefs drift apart. Either way, the survey gaps reflect what partisans actually believe.

But there is an increasingly popular competing interpretation. Perhaps some of the divergence arises not from how partisans perceive the world, but from how they choose to answer survey questions. A survey is a low-cost opportunity to signal allegiance to one’s team. When a pollster asks a Republican whether Obama had Trump’s phones tapped, the answer may function less as a factual report than as a form of political cheering. It feels good, it signals loyalty, and it costs nothing. Specialists call this “expressive responding.”

These two broad interpretations paint very different pictures of the electorate. On the sincere-belief interpretation, partisans are genuinely mistaken and/or genuinely divided by the information environments they inhabit. On the expressive-responding interpretation, they know more than they admit, and the apparent epistemic crisis is partly an artifact of the way we measure “beliefs”. Surveys mistakenly assume that people report what they think. If we instead treat surveys as low-cost opportunities for political signaling, then we cannot trust our surveys to tell us what people think.

Paying for the Truth

How can these interpretations be told apart? One answer is money. If survey answers are cheap talk, then we can make such talk costly. By paying people for correct answers, we can see whether partisan gaps on factual issues shrink. Someone who abandons a claim about illegal votes for sixty cents might not have been firmly committed to it in the first place.

This pay-for-accuracy paradigm has produced a decidedly mixed body of evidence. In two influential studies from 2015, John Bullock and colleagues, as well as Markus Prior and colleagues, found that modest payments substantially reduced partisan gaps in factual responding. But later work pushed in the opposite direction. Adam Berinsky found that incentives had little effect on politically charged rumors, and Erik Peterson and Shanto Iyengar found that partisan gaps in knowledge about COVID-19 persisted even when incorrect answers were financially penalized. A recent meta-analysis of 44 studies by Matthew Graham estimates that incentives reduce measured partisan bias by about 25 percent on average. So there is something to the cheerleading picture, but much disagreement survives payment.

Almost all of this work, however, reports a single average effect — how much incentives shrink the partisan gap across all respondents. That average hides two things our results turn on. It collapses Democrats and Republicans into one number, so it cannot show whether payment moves one party and leaves the other untouched. It also collapses questions that favor different parties, so it cannot show whether payment raises accuracy on some items while lowering it on others. Our study was designed to pull these apart.

Our Experiment

We recruited 888 U.S. participants and asked them to evaluate twelve factual political statements as true, false, or “don’t know.” Each statement had what we call a “partisan valence,” meaning that the correct answer favored one party. “Global warming is happening and is mainly caused by humans” is true, which fits the Democratic worldview. “Illegal immigration increased under President Biden compared with the Trump administration” is also true, which fits the Republican worldview. Half of the participants answered without incentives (apart from a flat participation payment). The other half received $0.60 for each correct answer and twenty cents for admitting ignorance (“don’t know”).

All twelve statements are shown below, grouped by partisan valence and marked with the correct answer.

Without incentives, the partisan gaps were enormous. On Democratic-valence questions, Democrats answered correctly 62 percentage points more often than Republicans. On Republican-valence questions, Republicans answered correctly 44 percentage points more often than Democrats. (These are model-based estimates that adjust for differences across questions; a simple average of the individual questions gives somewhat smaller gaps, for reasons I explain below.) What this means is that neither side is better informed in any general sense. Each side is strikingly "knowledgeable" about facts that flatter themselves, and strikingly "ignorant" about those that do not. Below are the results by each question item.

(A note on how the numbers are calculated. The dot plot above reports raw accuracy on each individual question—the simple share of each group that answered it correctly. The bar chart below, together with the partisan gaps I quote in the text, comes instead from a statistical model that pools across all twelve questions while adjusting for the fact that some questions are much easier than others and that the two sides did not face the same number of questions. These two approaches give slightly different answers. Average the dots for the no-incentive condition by eye and the partisan gaps come to roughly 46 and 36 percentage points, while the model puts those same baseline gaps at 62 and 44. I quote the model-based figures in the text because they are the ones the paper relies on.)

Three findings stand out.

First, and most strikingly, the incentives worked almost entirely on one side of the aisle. Republicans shifted their answers when we paid them; Democrats and Independents barely moved. This is the finding we did not expect and have not seen reported elsewhere. The existing literature treats accuracy incentives as a general corrective, a way to strip partisan performance out of survey responses whoever the respondent happens to be. When earlier studies find that payment shrinks partisan gaps, they present it as an effect that operates on partisans as such. Our data tell a different story. The gap narrowed not because both sides started to converge, but because one side responded to the money and the other did not. Any account of what incentives do to partisan responding now has to explain why they do it so unevenly.

Second, Republicans did not simply become more accurate. On questions where the correct answer favored Democrats, they became about 14 percentage points more accurate (from 39% to 53%). But on questions where the correct answer favored their own party, they became about 5 percentage points less accurate (from 69% to 64%). The incentives redistributed accuracy across question types rather than increasing it. The partisan gap narrowed for both types of questions, but through movements in opposite directions, and overall accuracy did not improve meaningfully.

Third, “don’t know” is not a neutral answer. Consider participants who failed to answer a Democratic-valence question correctly. They had two options: give a false answer or admit ignorance. Republicans overwhelmingly chose the false answer that flattered their party, while Democrats were much more likely to say “don’t know” (a difference of 34 percentage points). This makes sense once one recognizes that both answers can serve partisan purposes. For a Republican, asserting the congenial falsehood expresses loyalty; for a Democrat, saying “don’t know” avoids affirming an uncongenial claim. Even the admission of ignorance, it turns out, is political.

What’s Going On?

One natural thought is that incentives simply make people think harder. If that were the mechanism, accuracy should have improved across the board, for both parties and both types of questions. But it did not. Republicans actually became worse on questions favoring their own side.

The pattern fits the partisan cheerleading picture. On this interpretation, Republicans without incentives were engaging in expressive responding, giving the party-flattering answer regardless of its truth. That inflates measured accuracy on Republican-valenced items, where the flattering answer is the correct one, and lowers it on Democrat-valenced items, where the flattering answer is false. Payment then reduced the incentive to cheerlead. As a result, accuracy rose on Democrat-valenced questions, where the flattering answer had been suppressing correct responses, and fell on Republican-valenced questions, where cheerleading had been propping accuracy up.

But there is another possible interpretation that predicts exactly the same data. Rather than eliminating insincerity, the incentives may have introduced a different kind of insincerity. Perhaps Republicans inferred that the researchers were from the political left, guessed which answers the survey-takers would count as correct, and gave those answers in order to collect the money, regardless of what they believed. Call this “strategic responding.” Notably, this explanation works only if respondents perceived the research context as left-leaning, since otherwise giving the answer the researchers regarded as correct would simply amount to answering correctly. Our design cannot cleanly separate these two explanations. Distinguishing them would require manipulating the perceived political identity of the researchers themselves—and that is an idea for a follow-up study we are now planning.

Either way, one conclusion remains. The picture of “different factual realities” is too simple. A substantial portion of measured factual polarization likely reflects how people choose to answer, not what they perceive. That is unsettling news for anyone who takes surveys of political belief at face value. It may also be mildly reassuring news for anyone who fears that half the population is unreachable by evidence.

—

This post is based on a draft manuscript titled “Who Changes Their Answers and Why? Accuracy Incentives and Partisan Expressive Responding,” coauthored with Olaf Borghi and Irena Arslanova.