What good are non-replicable studies?

A recent project at the University of Virginia examined nearly 3,900 social science studies published in 62 journals from 2009 to 2018 and attempted to replicate 164 of them.

Only 49% were successful.

“Usually in research we’re making a major general claim,” said Brian Nosek, a psychology professor at UVA and head of the project. For a general claim to hold up, it must be replicable.

If over half of the studies aren’t replicable…what’s the good of the social sciences? Do they have any value, or are they just a load of crap gussied up to merely look like science?

As a former sociology student at Harvard, and someone whose work continues to rely on and intersect with the social sciences, I find this question paramount. As I’ve been mulling over it the past few days, here’s the conclusion I’ve come to:

Social sciences reveal objective truths about subjective experiences. Those truths need not be widely generalizable to be valuable.

Let me explain.

I find people are frequently surprised when they ask why I chose to study sociology, and I reply, “the methodology.” I did quite a bit of wandering through the social sciences before I settled on the path trodden by Emile Durkheim, Robert Nisbet, Ferdinand Tönnies, Robert Sampson, and many others.

In economics, I felt that the numbers failed to capture the full complexity of human experience. In anthropology…I felt like I needed at least a few numbers…(no shade to any anthropologists in the chat).

Sociology, with its methods that combined in-depth interviews, ethnographic study, quantitative data analysis, and visual mapping, all in the service of understanding human group behavior, made me fall in love.

It allowed me to capture the nuances of the lives and experiences of particular people living in particular places at particular times with vivid detail and to understand their worldviews with engaging depth. I wrote papers on how the campus environment shaped a first-year’s sense of belonging, how conservative women at Harvard understood femininity, how ownership and trust impacted the urban planning process in Chicago, and more.

I am unsure of how “replicable” any of those studies would be on different campuses, in different cities, or in different times. And I’m okay with that. In my view, it doesn’t make them any less true or insightful.

I’ll use one of the UVA study’s examples to illustrate my point.

A 2012 study found that extroverts experience closer bonds to their workplace, team, or school. When UVA researchers tried to replicate it, measuring first students’ personalities, then asking them how they connected to the university emotionally and cognitively, the link between extroversion and emotional connection didn’t emerge.

I’ll be honest, I don’t know the details of the original study, so it’s entirely possible that the science was always bad and no such connection ever existed. But I think it’s also very possible that, in the 10+ years since 2012, the evolution of people’s relationships and identity through social media (particularly for young people), as well as the shifting culture of what college means politically and economically, could also have changed the link the original paper observed.

For these reasons, the fact that these results weren’t replicable doesn’t shake my faith in the social sciences. Similarly, I wouldn’t be surprised if, despite Max Weber’s classical theories on the work ethic of Protestants, a study today found no distinction between the work ethics of Protestants and Catholics, or even the opposite (that Catholics work harder than Protestants). In my view, that doesn’t mean his theory was wrong at the time. It also doesn’t mean religion plays no part in how people conceive of and relate to their work (perhaps Catholics are found to work harder because they must ‘work’ out their salvation, meanwhile Protestants increasingly emphasize that ‘through grace alone you have been saved.’).

The beauty of objective truths about subjective experiences is that they are subject to change with the people. And the beauty of people — perhaps to the horror of those who try to quantify and define everything — is that they are always changing.

Perhaps replicability is not the ultimate measure of legitimacy that we think it is (at least, not for every study). We are trying so hard to make generalizable, blanket claims about all people; maybe that’s not what the social sciences are trying to show us.

Maybe they’re just meant to give us a beautiful, albeit limited, view into the experiences of our fellow human beings. Not despite their complexity, but in awe of it.

I think that’s still a beautiful and worthwhile pursuit. What about you?

Leave a comment

Share