Flawed Research is Not Hypothesis Generating

There is a prevailing error that shows up repeatedly in clinical research discourse: the idea that any observational study is “hypothesis generating.”

A well-designed study that is underpowered, narrowly sampled, or contextually limited can meaningfully generate hypotheses. A study that is biased cannot. That distinction matters.

Hypothesis-generating work should be defined by validity with imprecision or lack of generalizability. The estimate may be noisy, but the signal is directionally credible. A small, well-phenotyped prospective observational cohort with appropriate measurement and outcome ascertainment fits this mold: limited generalizability, wide confidence intervals, but internally coherent.

By contrast, a study with major, unresolvable sources of bias produces estimates that may be extremely precise, however, uninterpretable. When the direction and magnitude of bias are unknown or unstable, the results are not “hypothesis generating”. They are noise.

This is why a massive retrospective analysis of electronic health records data is not inherently more informative than a small prospective study. In fact, it is often less so.

Scale does not rescue bias. It amplifies it. Confounding by indication, measurement error, misclassification, unclear temporality, and differential outcome ascertainment do not attenuate with sample size. They risk becoming more precisely wrong.

One of the most problematic complications of end-stage dementia is trouble swallowing, known as dysphagia. These patients are at high risk of aspirating food into their airway, leading to respiratory distress, pneumonia, and death.

A common strategy to mitigate this risk is to thicken liquids. While this approach improves surrogate measures of aspiration, it has not translated into fewer clinically meaningful outcomes, such as pneumonia, in smaller trials. It also comes with downsides of poor palatability and reduced intake, raising concerns about dehydration. (For the brave soul, give the GeriPal #ThickenedLiquidChallenge a go.)

So there is a clear need for better evidence on whether thickened liquids actually help. Enter a 2024 JAMA Internal Medicine study comparing thick versus thin liquid diets in hospitalized older adults with Alzheimer’s dementia and dysphagia.

The authors executed a technically sophisticated retrospective cohort using multi-hospital electronic health record data of nearly 9,000 patients, including advanced adjustment methods. This is careful work. But the limiting issue is not effort or technique. It is the nature of the data itself.

I served as a peer reviewer for this study. At that time, I wrote:

Unfortunately, these limitations are too great to consider these findings hypothesis generating as the direction of the biases are large and unpredictable.

Here is a summary of my core critiques:

• Critical confounders omitted: Severity of dementia and dysphagia, arguably the most dominant determinants of both exposure and outcomes, are not measured.

• Exposure misclassification: “Thick liquids” was defined heterogeneously (>75% of days), with unknown adherence and no accounting for co-interventions, such as chin tuck maneuver that speech therapists teach caregivers.

• Outcome and covariate validity concerns: Reliance on unreliable administrative billing codes for key constructs like delirium, choking, and admission diagnoses.

• Unclear temporality: Due to lack of timestamps of exposure and outcome, diet changes may merely follow aspiration events, making reverse causation a highly plausible explanation.

• Internal inconsistency: Increased respiratory complications but decreased intubation in sensitivity analyses using an alternate modeling approach.

Taken together, these are not minor threats to inference. They fundamentally destabilize it. The relationship of cause and effect may not even hold. If it does, the direction of bias is potentially large and unpredictable. In the context of multiple large biases, the observed association cannot be mapped back to a plausible causal effect.

The Overreach

To their credit, the authors are unusually transparent about these limitations and explicitly caution against influencing clinical practice. That level of restraint is rare and commendable.

But there is still a conceptual overreach. The study was funded by an NIH R03 grant with the stated goal:

The findings from this proposal will be used to support the application for a large-scale clinical trial to prospectively evaluate the effects of dietary restrictions…

From this study, the authors conclude:

This cohort study emphasizes the need for prospective studies that evaluate whether thick liquids are associated with improved clinical outcomes in hospitalized patients with ADRD and dysphagia.

This framing and conclusion position the work as foundational, as laying the groundwork for future trials. That is reasonable in intent. But it hinges on the assumption that the findings provide clarity. They do not.

A study whose estimates are dominated by unmeasured confounding and ambiguous temporality does not meaningfully inform the design of a future trial. At best, it identifies that the question is important, which we already knew in this case. At worst, it suggests misallocation of scarce research funding by prioritizing misguided research directions.

Where Hypothesis Generation Actually Lives

If the goal is to generate hypotheses about thick versus thin liquids in this population, a better design is obvious: a dataset with careful phenotyping of dementia and dysphagia severity, standardized exposures, well-measured confounders, and clear temporality between diet order and outcomes. That dataset likely does not exist and would need to be prospectively assembled.

It would be small. It would be underpowered. But it would produce more interpretable findings. That is the tradeoff: imprecisely valid versus precisely invalid.

In this case, though, the hypothesis has long been established. What is needed now is not another attempt at hypothesis generation, but a more definitive and generalizable clinical trial.