TabPFN and TabICL are not everything

How do you pre-train a tabular model?

One answer is tabular foundation models like TabPFN and TabICL. They are prior-data fitted networks, an approach to tabular foundation models. I covered the PFN family extensively in this blog (start here). In short, TabPFN and TabICL are transformer-based networks, pre-trained on millions of synthetic datasets, allowing for in-context learning.

But what about other approaches to learning from some tables and transferring them to another table?

Besides PFNs, there are hypernetworks, cross-table transfers, and LLM-based approaches.

This post zooms out a bit, looking at these other tabular foundation model families.

Hypernetworks are pre-trained to “predict” or emit weights of a smaller model, typically a multi-layer perceptron (MLP), which is then used to make the actual prediction. The “prediction” step of the hypernetwork, therefore, replaces the training step of the MLP. And the actual prediction for your current data is then just regular inference with an MLP. Hypernetworks are often based on transformers (like MotherNet), but don’t have to be (for example, iLTM)

This family follows the more classic foundation model recipe with self-supervised pre-training across many tables, then fine-tuning on your current data. The pre-training task is usually self-supervised, for example, a masked reconstruction task, where some cells are hidden, and the model predicts them. Approaches in this family have in common that they learn to encode diverse tables into a schema-agnostic representation, and they are often transformer-based.

During pre-training, each table might get its own input/output adapters (featurizer and head) that are discarded for inference. Instead, to apply the model to new data, you attach a fresh set of adapters and fine-tune.

The family holds a diverse set of models, differing mostly in how to encode tables into a schema-agnostic representation. For example, some approaches also encode semantic information such as column names (e.g., CARTE), while others don’t (e.g., XTab).

The basic idea: take a table, turn it into text, then fine-tune an LLM in a supervised fashion. This reframes table prediction as a language problem to force it into the LLM scheme. I find that the weirdest family, as LLMs have the wrong inductive biases for tabular data, especially when the tables have lots of numerical values. On top of that, inference cost is relatively hefty. The LLM-based family may be most interesting for small tables with lots of semantic features, like free text features, categories, and telling feature names. This TFM family is the only one that allows zero-shot predictions, but can also do few-shot learning (aka In-Context Learning). An example is TabLLM.

Part of what brought me to write this post is thinking about the scope of my latest book project, Tabular Foundation Models.

The book is focused on just one of the families, the PFNs. So why not cover these other families in the book? Here is my reasoning for why I’m betting specifically on PFN-based models like TabPFN and TabICL:

• PFNs are currently state-of-the-art, even beating boosted tree models.

• Great open source availability and ecosystem.

• There is a lot of development happening in the PFN space.

• Many startups and companies are betting on PFN-based models.

• PFNs allow for in-context learning, no further training required.

• No fine-tuning required. But you can fine-tune if you want to.

• It’s easy to instill the right inductive biases, as you can include synthetic data in the pre-training.

So, to me, PFN-based approaches like TabPFN and TabICL hold the greatest promise for foundation modeling. That’s why I’m betting that they’ll end up as a new state-of-the-art paradigm in tabular modeling.