The Problem with Beta-Carbolines, Part II

There’s a joke among scientists that goes something like:

“A month in the lab is a great way to save yourself an hour in the library.”

It happens all the time: in the course of your usual work, you stumble across some intriguing and unexpected new phenomenon. You investigate, get some interesting results, maybe devise a theory around it, and you’re halfway through writing up what you’re sure is going to be an award-winning publication when you find out that—oh, oops! Turns out this is already a well-known thing in an adjacent field (or in your own field, 50 years ago), and the reason you hadn’t heard of it is that it’s a lot less exciting than you initially thought.

It’s gotten easier to avoid this embarrassment in recent years with the advent of Google Scholar, Pubmed, etc., but the story of the cluster headache that artefactual beta-carbolines and isoquinolines caused in neuroscience isn’t very well-documented online; I only know so much of it thanks to this Ancient Tome my dad gave me a few years back when I was old enough to start asking those questions.

And that’s why I wasn’t surprised to see the same story playing out in the headline of a paper which came out around this time last year, from a new lab at Harvard:

For context: While the intestinal microbiome is as dense and diverse as a jungle, vaginas are a lot less complicated.1 A healthy vaginal microbiome is typically dominated by one or two Lactobacillus species, and much of anything else is seen as a sign of trouble (although there are exceptions to this rule, especially when you start to look at non-Western populations).

Since Lactobacilli are best known for producing lactic acid, it’s widely thought that they serve to acidify the vaginal microenvironment, and thus help keep it free of pathogens: things like Gardnerella and Candida, common culprits in bacterial vaginosis (BV) and yeast infections respectively, have a much harder time growing at low pH. Over the years, studies have found that some vaginal Lactobacilli also produce things like antimicrobial peptides and hydrogen peroxide, which may contribute to this same end.2

This theory does a pretty good job at explaining what we observe when we study the vaginal microbiome in health and disease. For instance: taking antibiotics is associated with subsequent yeast infections, suggesting that stripping away the protective Lactobacilli leaves us susceptible to fungal overgrowth. A Lactobacillus-dominated vaginal microbiome is associated with lower pH and better outcomes in terms of fertility, likelihood of pregnancy complications, and disorders like PCOS. (This is as good a place as any to note that, although that’s the prevailing theory, there’s an alternative model which does about as well at explaining these observations, rooted in the little-known fact that the cells lining the interior surface of the vagina are decorated with proton pumps: the same molecular machinery that produces the strong acid in your stomach. Unlike in the stomach, though, the proton pumps in the vagina turn on primarily in response to estrogen. Since Lactobacilli are among the most acid-tolerant species in any microbiome, there’s a world where Lactobacillus dominance is a consequence, rather than a cause, of a highly acidic vagina. In this model, you’d expect hormonal issues to lie at the root cause of things like recurrent Candida, fertility problems, PCOS etc. with the connection to antibiotic use arising from the gut microbiome’s impact on estrogen metabolism. This is a significantly less optimistic model, in that the problem would be a lot harder to fix.)

In any case, you can see why someone studying vaginal Lactobacilli might be delighted to find that they apparently produce a bunch of interesting chemicals like beta-carbolines. If those chemicals do something consistent with our understanding of the organisms that produce them, like suppressing inflammation? Even better!

But naturally, given the cautionary nature of the tales in the Ancient Tome, I was suspicious from the title alone. By Figure 2, I was pretty sure that the entire paper is bunk.

The thing in the image above that set off the real alarm bells for me is the furfuryl group—the dangly little pentagon guy with an oxygen in it. This is almost identical to an aldehyde that forms when you dry out sugar-rich organic matter. It’s found in things like coffee, oats, and—in all likelihood—the kind of just-add-water dehydrated nutrient broth that they used to grow their Lactobacilli for this study. Also found in that powder? Tryptophan!

Mix them together and boil it down, and you’ve got a recipe for the chemical labeled BC3 up there. Some minor variations on the theme, like using the dipeptide Trp-Val in lieu of tryptophan, or 5-furfural (another compound that can result from cooking certain sugars) for the aldehyde, and it’s easy to see how you get the whole zoo of compounds. Extract 14 liters of broth in ethyl acetate, as reported in their Materials & Methods (yielding acetyl esters BC7&9), evaporate to dryness, and you’ve apparently got a cover story in a Cell-adjacent journal.

Interestingly enough, while some of these chemicals hadn’t been reported before, BC6 already had a name—perlolyrine—because it had been previously identified in a variety of heat-treated foods.

The authors nod vaguely in the direction of this possibility, acknowledging: “Spontaneous production of β-carbolines has been shown to be enriched under acidic conditions. It remains unclear whether lactobacilli production of β-carbolines is driven by an as-yet unidentified enzyme or spontaneous production or both”.

But knowing that they form spontaneously, it’s worth asking: are we sure the bacteria are even involved? Are we maybe just looking at Maillard products, here—like the brown crispy layer that forms on your toast? And in reading through the paper’s methods, I was amazed to discover: they didn’t actually check!

Reader, they did not have a negative control!

At no point did anyone on this team of 20+ Ivy-League students, staff, and faculty think to check whether these chemicals were there in the un-inoculated broth. (Or, if they did, they didn’t report it).

Instead, they tested these nine varieties of basically-caramel by exposing epithelial and immune cells to them in a petri dish. They found that perlolyrine had some anti-inflammatory (or, less charitably, immunosuppressive) effects.

Then they gave some mice genital herpes and smeared the stuff in their vaginas. It sort of helped!

Reader, what the fuck!

Fortunately, the study’s other headline finding seems to save the day: they actually detected a few of these compounds in the human vagina. Even better, people with bacterial vaginosis (BV) had less of them! That’s practically case-closed then, isn’t it?

Unfortunately, we don’t buy this either. Take a look at the figure where they report this finding:

First off: notice that the concentrations are in the picomolar range. That’s literally a thousand times lower than the lowest concentrations they used in their in vitro and mouse work. It’s a clue that something is off. A further hint: these compounds aren’t even present in most of their subjects, healthy or otherwise.

Fortunately, the authors posted their raw data in the supplementals!

Here, the samples are grouped by Nugent score, but—on a hunch—I sorted them by sample number, re-indexed them from 1, then plotted the reported concentrations.

Dropping the non-detects, the correlation coefficient goes up to ~0.9.

Somehow, beta-carboline concentrations are almost entirely a function of sample number!

This is consistent with a model where amino acids in the vaginal samples react with some aldehyde contaminant in the chromatography column, consuming it in the process to form the detected compounds. If the samples were run more-or-less in number order, you’d expect the highest concentrations of these products in the first few, steadily declining until they’re undetectable in the entire back half of the sample set.3

Funnily enough, this also explains the loose association between beta-carboline levels and BV status. Consider: it’s always easier to find healthy controls than disease subjects for a study; Nugent 0s are a dime a dozen. As a result, the low sample numbers—early recruits—are more likely to be healthy, i.e. have low scores. If beta-carboline concentrations are a function of sample number, and sample number is partially a function of Nugent score, it follows that beta-carboline concentrations would also be a function of Nugent score—although you’d expect it to be a much less robust correlation. And this is what we see.

It’s hard to imagine the authors didn’t notice this sample-number effect. Hard, but not impossible: If they had, you’d think they would make the raw data “available upon request” rather than just uploading it for all the world to see.

But charitable interpretations only go so far. Look at the line I added to the first graph in this section—the LLOQ. That expressly indicates: “below this line, we don’t trust the numbers”. Even so, those numbers get used in their analysis, to produce that two-star 0.007 p-value. Nobody does that without knowing, in their hearts, that they are doing something wrong. This shouldn’t have made it past peer review, but what else is new?

So I emailed the lead author, inquiring politely at first. Had they actually measured beta-carboline concentrations in the supernatants used in their animal work?

They had! But the results looked like the vaginal samples—picomolar, a thousand times lower than the concentrations necessary to produce an effect in vitro.

So they had this data, they had run the experiment, but they didn’t include it in the paper because it contradicted the narrative they were building. After a little more correspondence, I shared my biggest concerns, asking—somewhat apologetically—if they had noticed the sample-number effect, and explaining that something similar had happened to us back at Holobiome, when we were looking at N-acyl amides.

She replied that the samples weren’t run in numerical order, which may be true to some extent, but doesn’t change the fact that beta-carboline concentrations are 1000x more tightly correlated to sample number than to the biological variable they were studying. That demands an explanation.

I asked if they had checked the un-inoculated media for beta-carbolines. I got an out-of-office autoresponder. Fine, okay, I am annoying.

And if you’re a newly minted Harvard lab chief, who’s no doubt fought tooth and nail for this place at the top of the prestige heap, the thought of having to retract the first paper your lab ever puts out because it was wrong in some very obvious ways probably makes you want to curl up in a ball and scream.

I didn’t want to do that to her. But I wrote to the journal, asking if they’d publish a response or a reanalysis if I submitted one, maybe a letter to the editor kind of thing. I long for the days when scientists would beef openly in the pages of scholarly journals—arguing and rebutting one another in new open letters with each subsequent issue, letting the readers follow along and decide who to believe. Unfortunately, those uncivil days are gone; Cell Host & Microbe had no interest in publishing such a thing. Understandable! If something really shoddy can slip by your review process, you must not be a very high-status publication. Best to keep up appearances.

I wrote the response anyway because all this went down while we were bumming around Amsterdam on our honeymoon, meaning to send it to another journal, but never got around to it. Maybe I’ll put it up on BioRxiv…but the final section of it follows:

This kind of “fuzzy fraud”—like the selective omission of certain results which contradict an otherwise-clean narrative—is all too common. It is at once understandable, given the incentive structures in modern academia…and unforgivable, being a violation of scientific principles at the most fundamental level. When one begins, in the words of Sir Arthur Conan Doyle, “to twist facts to suit theories, instead of theories to suit facts”, one has become an enemy of the truth.

We are beginning to see a reckoning with the problem of fraud in biology, as evidenced most recently by the demolition of Eliezer Masliah’s body of work and reputation. However, the Masliahs of the world, the unabashed fabricators and photoshoppers—their work represents the mere tip of the Iceberg of Bad Science. The subsurface bulk of it is composed of more-or-less honest mistakes by well-intentioned people, glued together by financial pressure and willful ignorance.

There is a pithy saying, fashioned after Occam’s Razor, which goes something like:

“Never attribute to malice that which can be explained by mere stupidity.”

While charitable in a sense, it doesn’t prove very useful in the face of malice, or even in the gray land of plausible deniability. The altogether less pithy “Hubbard’s Corollary” goes:

“Never attribute to malice or stupidity that which can be explained by moderately rational individuals following incentives in a complex system.”

And this is the crux of it—but as banal and impersonal as this sounds, it nevertheless represents a moral crisis at a personal level. Biology is the frontier of medicine, and in that capacity biologists are entrusted with vast sums of public funds, with the expectation that we’re doing the best we can to cure everything from cancer to bacterial vaginosis. In that sense, even supposedly minor “sins”—like dropping an iffy outlier that you maybe shouldn’t have, or using the statistical test that gives the best p-value, even if it’s not the most appropriate—are ultimately a choice to prioritize advancement and prestige for yourself, borrowing against the suffering of people who are counting on you to help them.

Honoring that trust, as a scientist, is not served by tucking away the loose threads of a narrative, but by pulling and testing each thread in turn, in case it makes the whole thing unravel. It’s a dismaying feeling when it does, but it’s a step closer to the truth, and it’s an invitation to make the next version less like fabric and more like chain mail: crafted one careful loop—one experiment—at a time, each ring interlocking with all those around it such that, should one link fail the test of time, it does not break the pattern of the whole.

And although it is cumbersome and difficult to make, such solid armor has this advantage over cloth:

You will never find yourself suddenly naked in the street, just because some little goblin saw a loose thread and decided to pull.

—🖖🏼💩

Stay tuned for Part III later this week, in which we’ll bring it back around to the schizophrenia hypothesis.