In Part 1, we introduced the 3 tabular foundation models and went through the settings that most affect their results.
In this part, we turn to the question that really matters in practice: how well they perform once we benchmark them properly.
To do that, I ran all 3 of them against CatBoost, the model that has dominated tabular data for almost a decade, across 12 datasets. I included CatBoost in 2 variants, one tuned and one with default settings, so that we can also see how much the tuning actually contributes.
Public leaderboards like TabArena already rank these models, but they tend to report accuracy on a single fixed suite of datasets. I wanted a closer look, one that also accounts for speed and calibration, including the tuning cost.
By the end, we will have a clear view of which model is most accurate, which is fastest, which produces the most trustworthy probabilities, and whether the gaps between them are statistically real or simply noise.
✅ Find the full benchmark, every dataset, and the code to run it in the AI Projects Folder (Project 35). Plus more cool projects inside!
The benchmark uses 12 classification datasets from OpenML, chosen across easy, medium, and harder sizes. We compare 5 models in total: TabPFN-3, TabICL-v2, TabFM, and 2 CatBoost variants (one tuned, one with default settings). For each model, we record accuracy, calibration, and running time.
One design choice is worth stating up front: the tuning time counts against CatBoost. In real projects, that time is something we have to spend, so it belongs on the clock alongside everything else.
We also save every prediction to disk. As a result, the metrics can be recomputed without a GPU, and the entire analysis is reproducible from the saved files. (The full code for this is in Project 35.)
Table 1 below is the headline result. Each row is a dataset and each column a model, with the best score in each row shown in bold. Ties are all shown in bold as well, since several datasets end in an exact tie.
Even a quick look reveals the pattern. TabFM holds the most bold cells, while TabICL-v2 and TabPFN-3 share most of the remaining ones. The 2 CatBoost columns are bold only where they tie with another model. On the volkert dataset, the difference is especially large: TabFM reaches 0.805, while both CatBoost variants sit near 0.67.
Macro-F1 and ROC-AUC point to the same ranking, so I keep the figures focused on accuracy, speed, and calibration. Find the rest of the tables in the companion notebook.
Counting wins seems straightforward, but ties complicate it. If we simply take the top score on each dataset, a tie is quietly awarded to whichever model happens to appear first in the table, which is not a genuine win.
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.