It's actually not all in the details

I wanted to share and discuss this recent preprint from Bo Wang’s group on introducing a foundation model for echocardiography. Nowadays, it seems like everyone is working on some sort of “foundation model” within some niche in medicine - be it cancer biomarkers, admission events, genomics, or imaging scans. However, there are a couple of important insights from this work that I think are particularly interesting in the context of thinking about the overlap between adult and pediatric medicine.

In the paper, the authors introduce a new foundation model called EchoJEPA for echocardiographic data. The significance of this (any really any foundation model) is that it can take echocardiography videos and turn them into vector representations (called embeddings) that are clinically correlated with the clinical picture captured in a given video.

For example, we might imagine that EchoJEPA would output similar embeddings for echocardiography videos that show similar pathology across different patients, and more distant embeddings for pairs of videos that show unrelated pathology. A very, very crude way of thinking about embeddings is to imagine them as abstract representations of different disease processes in a “mental model” - for example, your mental models of patients that have heart failure might be more similar and “closer together” than your mental models of a patient with heart failure and another with a cardiac embolus. The exact details of individual patients might be fuzzy, but you might have a general gestalt about how patients with these diagnoses might be different from and similar to on another.

EchoJEPA essentially formalizes this intuition in computer science lingo, and applies this intuition specifically in the context of echocardiography interpretation. The reason why I think EchoJEPA is so interesting is that it’s one of the first medical foundation models (at least that I’m aware of) that uses JEPA as its backbone training method. For those that are unfamiliar, JEPA and its follow-up derivates were first introduced by Yann LeCun’s group in 2022 as an alternative research direction from the generative models that everyone had (and have still) been focusing on. The main idea behind this method was that model predictions are performed directly in the embedding space. This approach essentially shifts the focus from perfectly matching every detail in the inputs to matching the “main ideas” of the inputs according to the embeddings.

If this sounds complicated, you’re not wrong: there’s a lot of math and theory that goes into the formulation of JEPA. Going back to our earlier example, one way of thinking about it is that traditional generative models would try to learn how patients are similar to one another by matching every single detail: we might try to learn the average age, gender, risk factors, and entire past medical history for patients with CHF. But not every detail is important! For example, I probably don’t care whether or not patients with CHF exacerbation had a lap chole two decades ago, and so it doesn’t make sense to try to go digging for remote surgical histories and match them between patients. JEPA instead tries to learn a consistent abstract mental model of CHF patients, focusing on the important details while glossing over components of a patient’s history that aren’t as relevant.

In other words, we’re compressing clinical information - a behavior that emerges implicitly because anytime we try to learn embeddings that are predictive while also being lower-dimensional. This forces JEPA models to focus on the details that actually matter for clinical downstream tasks.

Because EchoJEPA forces us to only focus on the pertinent parts of an echocardiography that actually have clinical utility, the authors show that the resulting model also generalizes better to pediatric echocardiograph interpretation even when trained solely on adult imaging data. Specifically, the example they give is that the error in the predicted LVEF is lowest using EchoJEPA when compared to existing baseline methods for pediatric patients.

Obviously, there are some limitations to this study: it’s unclear what’s the clinical significance of the differences in LVEF predictive performance between the different models, and I’d be hesitant to say that one foundation model is “better” than another based solely on a single clinical task without any primary clinical endpoints. I’m more interested in the overall direction of studies like this - and what it might mean for how AI might apply to clinical medicine in the future.

I’m no expert in echocardiography and don’t particularly know what this study means for that area of research in particular. Nonetheless, the reason why I think this study is interesting is that by using JEPA to implicitly focus on the higher-level features that are conserved across different scans, the authors show how the learned model representations are able to generalize better to different patient populations - including pediatrics.

A lot of my clinical training so far has been focused on all the ways that adult and pediatric medicine are different - different incidences of disease, different treatment modalities, and different disease burden. However, tools like EchoJEPA might help us better understand how some aspects of these two clinical specialities might be more similar than we might think. By learning to focus our attention on higher-level patient features, as opposed to every single detail available in an image or health record, we might be able to arrive at more generalizable mental models of patients that translate clinical insights from one patient population to another.

If you enjoyed this content, consider subscribing! I’m an MD-PhD candidate at Penn and hope to share advancements in AI research as they pertain to internal medicine and pediatrics.