TabFM minus the hype

Google Research published TabFM, a new tabular foundation model.

The AI influencers found out, and social media is now awash in TabFM posts.

Let’s talk about what TabFM is and isn’t, without the hype.

TabFM is a tabular foundation model like TabPFN, TabICL, and TabDPT.

More technically, TabFM is a transformer-based neural network pre-trained on hundreds of millions of synthetic datasets. TabFM makes predictions via in-context learning: there is no classic training step, but the training data is provided during inference time and serves as the context to predict the test data. It can do both regression and classification (up to 10 classes).

If you want to learn more about tabular foundation models in general, read my TFM series:

TabFM follows in the footsteps of especially TabPFN and TabICL, which the authors note in their research announcement post. It re-uses architecture elements of both:

• For example, TabFM uses alternating row and column attention, as e.g. TabPFN 2 did.

• TabFM uses row compression and performs ICL over the compressed rows, as TabICL does.

Also, TabFM is pre-trained on hundreds of millions of synthetic datasets generated with structural causal models, just like TabICL and TabPFN. However, not much is known about pretraining and prior, because the published code is just the inference code, and there is no white paper or paper published, just an inference code repo, the research announcement post, and the Hugging Face model (weight) release.

Being a tabular foundation model, TabFM inherits the “standard” pros and cons of modern foundation models: No tuning needed; highly performant; slow inference; need to provide training data at inference time; and so on.

So, TabFM is not the first tabular foundation model, nor the last. What is all the hype about?

One of the reasons for the hype: TabFM climbs to the top of TabArena, a benchmark for (primarily) tabular foundation models. If you want to learn more about TabArena, I've got you covered:

TabFM’s entry is not yet reflected in the live leaderboard at the time of writing, but results are waiting to be included in a pull request. These results are certainly impressive.

So, should we all be using TabFM now? I won’t be, for now, for two reasons.

TabFM is larger than the other tabular foundation models. For example, it defaults to 32 estimators, when most other TFMs have a default of 8. Also, many aspects of the architecture are scaled up: TabICL contains 4 CLS tokens; TabFM contains 8. TabICL has 12 ICL transformer blocks; TabFM has 24. TabICL has an embedding size of 128; TabFM has 256. In many ways, TabFM is larger than the current 2nd-generation (TabICL v2.0; TabPFN-3.0).

This scaling up and the alternating row and column attention come at a price: Inference is slower than for the other TFMs. The maximum number of features is 500.

The bigger issue no one talks about is the license. If you go to their GitHub repo, you see an Apache-2.0 license. Great news, right? But that repository only contains the inference code. The weights are downloaded from Hugging Face and come with the tabfm-non-commercial-v1.0 license, which prohibits commercial use. In my understanding, if you want to use it commercially, you would have to contact Google and ask for permission.

For new TabFM users, I don’t think it’s obvious: At the time of writing, the non-commercial license is not mentioned in the README, and the weights are downloaded silently from Hugging Face. See also this GitHub issue. I hope Google addresses this; otherwise, users might unknowingly violate the non-commercial license.

All in all, TabFM's performance seems impressive, and I am looking forward to seeing how it holds up in other benchmarks. The model seems to be geared towards performance, at the cost of inference speed. The non-commercial license makes it less attractive for me.