If you enjoy my writing and want to support viewpoint diversity in academia, you can give me a tip! Or please simply, like, share comment, or subscribe:

Subscribe now

Just as I was receiving peer-review feedback on a manuscript I submitted to a psychology journal, demanding that I add “more theorizing” to the manuscript, published his essay Hasty Theories and put words to the skepticism I had been feeling lately about the “theory crisis” in psychology.

Inzlicht asks whether, in actuality, psychology has too strongly internalized the idea that theory will fix everything, and we now hastily and carelessly shoehorn theory into everything. This is perfectly exemplified in my peer-reviewers’ call for greater theorizing in my manuscript.

My study was the first in my little research niche to use the methods and analysis combination that I used. I used a sample size that, while large enough and justified by my power analysis, should probably be replicated in a larger sample. My study mostly found null results, which you can’t really meaningfully theorize about (absence of evidence not being evidence of absence). And so on. There are many reasons why theorizing shouldn’t be the primary goal of my study. In fact, I should be actively discouraged from theorizing (essentially speculating) too much based on my study.

As Inzlicht says:

Our top journals actively encourage this theoretical gold rush, demanding that every paper advance or propose a new theory. Try submitting a careful descriptive study and watch reviewers ask: "But what theory does this advance?" As if documenting something interesting about human behaviour isn't valuable unless it moves some theoretical ball forward.

I don’t mean to imply that theory is unimportant. But when your celebrated theoretical work turns out to be an elaborate castle built on data that was tortured into compliance, you get humble real fast. Perhaps rushing to theorize—before understanding what you’re explaining—just to get published in top journals is not ideal.

He describes how theorizing too early and too often just results in confirmation bias, especially when the theory advances ideas and worldviews which we find “socially compelling” or desirable:

We all fall prey to confirmation bias, even those of us who study it for a living. Researchers become so invested in their theoretical hunches that they'll keep tweaking experiments, massaging methods, and reframing results until they get what they expect, essentially bending their methods to confirm their pet hypotheses…

And if the theory is appealing—because it is intuitive or elegant or socially compelling—it can be seductive, discouraging dissent and surviving on its rhetorical appeal or supposed importance even as the evidence starts to look threadbare.

This rush to theorize, combined with the ideological homogeneity of social science, has meant that very often theories are simply hastily constructed narratives used to support the worldviews dominant among social scientists. This was seen in the theory of stereotype treat:

Imagine an alternative timeline where these effects were discovered without the grand theorizing. "Huh, some women seem to perform slightly worse on math tests under certain conditions when we remind them of negative stereotypes. Weird. Wonder if anyone else can find this?" If we'd approached it as a curiosity rather than a revolution in understanding inequality, maybe we would have noticed how fragile the initial evidence was. Maybe we would have demanded better replications before theorizing and building entire interventions around it.

But no.

We canonized these theories before we rigorously tested them. And, once canonized, theories stop serving truth and start serving the reputations of their creators.

Inzlicht correctly notes that building theories on questionable research is doomed to fail. Methods, measurement, validity, and accurate description must come first— psychology is prematurely trying to fix and generate theories before we have the foundation upon which theories must be built.

What would psychology look like if we took description seriously? For one, we'd be more patient before proposing new theories. Instead of racing to explain every interesting finding, we'd spend time mapping the phenomena first. When does an effect occur? How robust is it? What are its boundary conditions? These aren't trivial questions.

Kurt Lewin famously said, "There is nothing so practical as a good theory." But we've rushed to theorize, resulting in theories that are flimsy, trite, or false.

I propose canonizing another, less pithy line, this time from Solomon Ash (via Rozin): “Before we inquire into origins and functional relations, it is necessary to know the thing we are trying to explain.” In other words, description must precede explanation.

Below I will outline two major obstacles to theory generation in social psychology specifically that I don’t often see discussed.

The Underdetermination of Theory

The underdetermination of theory by evidence is a problem outlined by philosophers of science who point out that, very often, the available evidence is insufficient to definitively determine which of many competing theories is the correct one.

Very often, competing theories (and a whole host of yet un-proposed theories) could all explain some set of observations equally well. Additionally, even testing theories and subjecting them to falsification is often not conducive to deciding between theories, because when a prediction or hypothesis not supported, it is unclear whether the central hypothesis was false, such that it was responsible for the falsification, or whether any number of auxiliary assumption was falsified (say assumptions around manipulation effectiveness, measurement validity, etc.) and this caused the negative result. Unambiguous falsifications are very often almost impossible.

This is all true of science in general, but is even more of a problem for social sciences, in which the phenomena under study are far from being mechanistic and reliable, and our studies are built on a huge patchwork of quite dubious assumptions.

Human emotion and behavior is complex, contextual, and subject to multifactorial causation. Additionally, in psychology, our measures are usually indirect measures of unobservable “latent constructs”. We are unable to directly measure these constructs, so we have participants self-report their own “measurement” of the constructs through presumably accurate self-perception. These constructs are often fuzzy, ill-defined, overlapping, and tainted by hidden normative judgements (almost always of a left-wing bent). Experimental manipulations on such latent constructs are often “fat handed” and highly ambiguous.

Results from these studies very often show weak relationships or small effect sizes, such that using these results to make any sort of conclusive claim about some causal mechanism or theory explaining the phenomena is not valid. Very often the most we could say is something like, “this very ambiguous, fat handed intervention slightly increases the chances that some people might slightly increase how much they say they feel X, at least according to our fuzzy definition of X, which we measured using this hastily created scale (but α = > .8, so we’re good!)”. Likewise, supposed falsifications of a theory are equally ambiguous, even using so-called “severe tests”.

The underdetermination of theory by evidence shows that one could create almost any theory to explain some set of data, with limited useful ability to choose between different competing explanations. Thus, theory generation, as noted by Inzlicht, becomes a competition between researchers to create the most interesting, socially desirable, or otherwise appealing explanation—amounting to, essentially, a popularity contest.

Now, I love social science; it’s fun and interesting. All of the ambiguity gives plenty of opportunity for exploration and discussion. And despite the reasons for skepticism I think it is possible, but very difficult, to advance psychological knowledge.

However, when there is an artificial and hasty drive for theory construction under such conditions of underdetermination, and a popularity contest for theory selection, one aspect of the social science field really comes into play:

Ideological Homogeneity

The ideological homogeneity of social scientists actually bookends the problem discussed above, around underdetermination.

If you enjoy my writing and want to support viewpoint diversity in academia, you can give me a tip! Or please simply, like, share comment, or subscribe:

Subscribe now

As I have written elsewhere social scientists, like all humans, are often tribal epistemologists. As Clark and Winegard note: people (including scientists) “do not, on average, treat ideologically congruent and incongruent information similarly; rather, they engage in ideological epistemology such that they treat congruent information more favorably and charitably”.

Very often ideological biases get built in to instruments measuring constructs in psychology (see “Scale Creation, Use and Misuse: How Politics Undermines Measurement, In: The Politics of Social Psychology.”). These biases also manifest in the design of studies, the interpretation of results, the appraisal of competing explanations, and the generation of theories. Then, in an ideologically homogeneous field, this flawed and ideologically driven research doesn’t get pulled up in peer review. What results is a body of extremely flawed, incomplete, ideologically biased research, all of which only makes the aforementioned underdetermination and ambiguity worse!

In this environment, once the ‘theory popularity contest’ starts, what you have is a bunch of scientists (humans: tribal epistemologists) who share the same biases, priors, and ideology, apprising ideologically tainted research which does not contain enough unambiguous evidence to actually allow even an unbiased body of researchers to be able to discriminate between theoretical explanations.

What you get are endless theories about “stereotype threat” or “minority stress” or “system justification”, all of which serve primarily to massage and support the worldview of progressives rather than to validly explain some phenomenon. You get interventions which don’t work, and results which don’t replicate. Or you get theories which purport to explain social or cultural phenomena, which look like this:

And then once these theories proliferate, once social scientists buy-in to a theory (again, usually because it affirms their progressive worldview), it’s all they can see, and they often—consciously and unconsciously by their own ignorance (as many social scientists are not scientists at all)—engage in motivated research with questionable research practices and ignored alternative explanations in order to erroneously reify that theory and advance their progressive worldview rather than advance knowledge.

Trying to improve rigor in this kind of environment won’t work

Inzlicht ends his essay calling for a patient approach to theory generation. We need to first improve the descriptive rigor and measurement underlying our studies before we can even come close to generating accurate theories:

What would psychology look like if we took description seriously? For one, we'd be more patient before proposing new theories. Instead of racing to explain every interesting finding, we'd spend time mapping the phenomena first. When does an effect occur? How robust is it? What are its boundary conditions? These aren't trivial questions.

Kurt Lewin famously said, "There is nothing so practical as a good theory." But we've rushed to theorize, resulting in theories that are flimsy, trite, or false.

I propose canonizing another, less pithy line, this time from Solomon Ash (via Rozin): “Before we inquire into origins and functional relations, it is necessary to know the thing we are trying to explain.” In other words, description must precede explanation.

The path forward isn't to choose between rigor and theory; it's to use rigor and precision in service of better theory. Sometimes the most productive thing we can do is slow down, to ensure that we are observing carefully and describing what we actually see. Better theories through patience, not speed.

Look, Kruglanski's not wrong that we need good theories. But rushing to build them before we understand what we're explaining? That's how we got into this mess in the first place. Let's slow down and do this thing right this time.

I would argue, as do others, that slowing down and “doing things right this time” should be the goal, more than compulsive theorizing. Strong descriptive work is more important.

And doing things right this time is only possible if we improve viewpoint diversity in the social sciences.

If you enjoy my writing and want to support viewpoint diversity in academia, you can give me a tip! Or please simply, like, share comment, or subscribe:

Subscribe now