You Should Know About Surveys to Write About Surveys

The Argument is a very fun new publication. They’re doing the kind of data-driven journalism that I enjoy reading. One thing I especially enjoy is they often do original surveys to try to settle public questions. I love that. But I have noticed they often do a sort of bad job with the surveys. Here, we will look at one new article out today, by Maibritt Henkel .

My thesis here is going to be simple. The Argument has done an admirable thing trying to incorporate data into public-political-vibes reporting, they’re sincerely trying hard to do good survey work, but at the end of the day if you aren’t a survey person, you just shouldn’t put pen to paper about surveys. In this case, The Argument has fundamentally mischaracterized their own results in a way that reveals an elementary misunderstanding you’d be disabused of in the first 30 minutes of any serious study on how to use survey data.

The Argument uses a company called Verasight. Verasight is a bit of an up-and-comer in polling. They claim to have higher respondent quality than other similar low-price pollsters. This is similar to Prolific’s marketing, if you’re familiar. The key point is Verasight manages their own panel and doesn’t use 3rd party vendors. People vary in their opinions of this approach. On the one hand, this allows stronger quality control. On the other hand, this massively reduces your sampling frame. Verasight pulls from a few hundred thousand possible respondents; other companies pull from many millions. As a result, you are highly constrained on how you can weight variables, because Verasight’s sampling frame just doesn’t have some populations. The takeaway here is you get “good” respondents but relatively non-diverse respondents.

Likewise, Verasight supplements their online recruitment with address-based sampling. Old hands at surveying know this is “gold-standard” practice… except not really. Modern response rates on address-based sampling are insanely low and highly non-representative, such that address-based sampling can help you avoid bogus respondents but it can no longer plausibly claim to give you representative respondents.

Finally, Verasight has some good practices around identity verification and managing respondent attention. On the whole they probably do above-average in terms of quality, and their internal evaluations do claim to outperform other online survey companies and even some probability samples.

The questions The Argument asked are also reasonable. Nothing about these questions raises red flags about survey question design:

Reasonable questions in a decently plausible method with 3,000 registered voters is a good survey. I belabor this, because the point is, The Argument is doing it right, trying to hire a reputable company to do good work. Their questions are apparently vetted by an expert panel as well. All great.

Unfortunately, the challenges with surveying don’t stop at the survey.

Look at the graph above. It shows that 29% of men strongly disagree that men should be breadwinners, vs. just 11% strongly agree.

Does that support this claim?

One explanation for what is going on here is that men are answering out of solidarity with women — or, more cynically, a desire to give the socially desirable response — rather than pure self-interest. They reject the traditional division of labor into male breadwinning and female caregiving because they understand that sharing these responsibilities is good for women.

And yet, when we asked men what was more important to them — career success or getting married and having children — 44% of men prioritized family while 41% prioritized work, around the same shares as women. Among conservative men, the share prioritizing getting married and having children was even higher.

Obviously, no!

They did not ask men “Would you rather earn more than your wife or less than your wife?” They asked men “Should men be breadwinners?” The first question is a question about preference, the second is a question about norms.

It’s very easy to imagine that most men oppose the idea of a general norm of male breadwinners, yet most men might also prefer to out-earn their wives, or even have a stay-at-home wife. There’s no contradiction at all between these views. That which people espouse as general norms need not also be espoused as personal preferences. “I don’t think women should have to be 2nd class workers, but for my own family, it works best if I’m the primary earner” is in fact an extremely normal and common view to hold for many men!

Consider the claim they hope to rebut:

Some see the rise of the female breadwinner as a bad deal for men. Women may have gained independence, but they have done so at the cost of men’s sense of purpose. “Men are more likely to flourish when they see themselves as providers,” wrote Brad Wilcox, a University of Virginia professor and author of the book Get Married: Why Americans Must Defy the Elites, Forge Strong Families, and Save Civilization, along with co-author Maria Baer. “Men do better when they feel needed — and being a reliable breadwinner makes them feel needed.”

Does the survey above rebut this claim? No, not at all! Very obviously, W. Bradford Wilcox is referring to a personal preference. Brad did not claim “Men do better when they feel that women generally are inferior — and seeing other men be breadwinners makes them feel needed.” He made no claim about what kind of general norm men might espouse. He made a claim about what they feel in their own personal lives, a topic not surveyed here.

On the question of what men actually personally want, we have a very great deal of evidence that, indeed, men tend to be happiest when they make essential contributions to family economic wellbeing. This often but not always means being the primary earner, sometimes by a very lopsided margin, sometimes as sole earner. Likewise, we know that the share of women who prefer to stay home with children vastly exceeds the share for men. Brad’s claim is not rebutted and, in fact, remains enormously plausible.

To support the claim, Maibritt Henkel shows this graph:

And she says:

On the right, there is a sense that feminism has forced men into egalitarian relationships they do not want. Traditional marriage gave men a clearly defined role: to provide the primary income to their families. They would like that script back because without it they are in crisis.

Conversely, the left has tended to frame men relinquishing the breadwinner role as a moral imperative. There’s a sense that cheering on their wives’ careers and taking on child care is the least they can do.

Neither framing captures what our polling indicates: that changing gender norms around who spends time at home and who spends time at work might actually align with men’s own interests. Indeed, a majority of fathers say they spend too little time with their children, mostly due to work obligations.

Does any of this follow at all?

If I had to choose, of course I would say “Getting married and having children,” yet I’m the sole earner for my family. And guess what? Being economically successful is a key way men achieve marriage and children! Women continue to value men’s economic productivity extremely highly in relationships. When men become unemployed, their wives become much likelier to file for divorce, even though that divorce will make everyone involved even poorer. Perhaps the unemployed man became moody and hostile— but we needn’t pretend like women don’t find male idleness unattractive nearly to the point of repulsion! Witness the discourse around male-typical hobbies!

Paradoxically, when a man who says “Family is extremely important to me” probably has higher career ambition than a man who says “Family is not very important to me,” whereas for women, the relationship is far less clear, maybe even negative. This is why, for men, there is a strong positive gradient between income and fertility, whereas for women, it’s neutral to negative.

More to the point— for some men, the question asked is very optimistic! “If you could choose…” implies you’re guaranteed what you choose. I bet a lot of the men choosing family in the survey are, in real life, single and childless. Singleness and childlessness is more common among men of a given age than women due to age gaps in relationships, higher prevalence of men with multiple partners than men with multiple partners, and higher prevalence of women with interests in same-sex relationships.

This all adds up to me saying: it’s totally unreasonable to jump from “men value family” to “men want their wives to be their breadwinners.” Those statements could each be true or false but they have no relationship to one another at all. If you’ve read much about the rise of female breadwinners, you know the stay-at-home men are, very often, not involved and attentive fathers, but actually just deadbeats.

Consider this graph:

It’s very cool. This kind of survey design, where you ask respondents to guess what other respondents say, is becoming very popular, for the very good reason it allows you to produce graphs like this.

But what does this tell us? Does it tell us…

As in our February poll, there is little evidence in our survey data to suggest that men see traditional gender roles as more favorable than modern egalitarian ones. As Jerusalem Demsas recently pointed out, strong baseline support for gender equality is an underrated fact of our modern political landscape: Around 81% of both male and female respondents agreed that women are equally suited to leadership. Among male respondents who voted for Donald Trump, the share was lower but still 72%.

No! It does not tell us anything about “gender roles”! Traditional gender roles aren’t surveyed here at all!

What it tells us is male motivations are broadly misapprehended. A lot of men internally perceive themselves to be motivated by family, but externally they are perceived, by men and women alike, of being motivated by career ambition. It’s as simple as that. People impute good motives to themselves and bad motives to other people. When I log extra hours, it’s a valiant breadwinner providing for his family. When he logs extra hours, is a careerist creating a striver culture nobody can keep up in. The fact that people grade themselves favorably has been demonstrated time and again in many surveys.

This explanation is staring us all in the face here:

Whereas The Argument interprets choosing family as “choosing egalitarian gender roles,” in fact liberal men were less likely to prioritize family!

Prioritizing family just is a conservative value! We’ve been here before, folks! I mean seriously, look at this:

Trump-men favor family 57:32, but think men generally favor career 65:24. Harris men actually do favor career 48:33, and think men generally favor career 72:15.

It’s obvious that favoring family, and thinking other people may favor family, just is a conservative thing, so citing favoring-family as evidence of egalitarianism is just nutso.

Finally, I just have to note, the final header, “There has never been a better time to be a father,” is essentially a statement devoid of evidence.

I like The Argument and wish them well. I would like their style of engagement to succeed on the left. As a result, I desire for them to do a good job making their arguments.

They did not do a good job here, and the reasons was just a misunderstanding of what they were surveying. They assumed that their questions about norms could be interpreted as questions about preferences, and then imposed their own notions of what those preferences might mean (family-preferencing=staying home, etc).

Survey respondents are responding to the questions we ask them. We have an obligation as users of those responses to treat their responses with a modicum of seriousness and credibility. That means not reading in things they didn’t say. The Argument did not ask people about their personal preferences for the gendered division of labor in the household. Maybe they wish they had. But wishing does not make it so, and as a result, they instead egregiously misinterpreted the respondents they actually did get, in order to make a claim their evidence cannot support.