In our last benchmark, tabular foundation models outperformed fully tuned CatBoost on classification.
The result was clear on both accuracy and calibration, but it left one question open: does the same hold for regression, where the target is a number and the uncertainty around it matters just as much?
So I ran the test, and this time the setup has 2 new elements:
First, we tuned CatBoost on the GPU to give the trained model a fair chance.
Second, we added Nori-30M from Synthefy[1], a small open-weight foundation model built only for regression. It turned out to be one of the most interesting models in the benchmark.
By the end, we will know which model is most accurate, which is fastest, which gives the most reliable uncertainty, and whether the gaps between them are real or just noise.
✅ Find the full benchmark, every dataset, and the code to run it in the AI Projects Folder (Project 36). Plus more cool projects inside!
The benchmark uses 12 regression datasets from OpenML. They range from small to large, and several have categorical columns.
We compare 6 models: TabPFN-3, TabICL-v2, TabFM, Nori-30M, and 2 CatBoost variants, one tuned and one with default settings. For each model, we record R², error, calibration, and run time.
Our main metric is R², not RMSE. RMSE is the number most people expect, and TabArena reports it too. But RMSE uses each target’s own units, so we cannot compare or average it across datasets with different scales. R² always uses the same scale. That makes it the regression version of accuracy, and the right choice for a ranking.
We also count the tuning time against CatBoost, as we did in the classification benchmark. In a real project we have to spend that time, so we should measure it.
One dataset is worth a note, because it exposes a real limit. Mercedes has 376 columns. These in-context models keep the entire training table in memory during prediction, so a very wide table requires a lot of memory. TabFM is the heaviest model here, so it ran out of GPU memory on Mercedes and could not predict at all.
The lighter models, including both open-weight ones, handled it without trouble. To keep the comparison fair, we set Mercedes aside and rank the models on the other 11 datasets, and we return to what this means for TabFM at the end.
Finally, we save every prediction to disk. So we can compute the metrics again without a GPU, and anyone can reproduce the whole analysis from the saved files.
Table 1 is the main result. Each row is a dataset, and each column is a model. The best score in each row is in bold, and ties are in bold too:
The pattern is clear. A foundation model wins every dataset, and neither CatBoost column has a bold cell. The gaps are largest on the harder datasets. On socmob, for example, Nori-30M reaches about 0.90, while both CatBoost variants stay between 0.66 and 0.70.
The 2 open-weight models do well here. Nori-30M wins on airfoil and socmob, and TabICL-v2 wins on abalone. So a top score does not require a paid license. RMSE and MAE give the same order, so we keep the figures on R², speed, and calibration.
You can find more results and detailed explanations in the companion notebook.
Counting wins looks simple, but ties make it harder. If we just take the top score on each dataset, a tie goes to whichever model sits first in the table. That is not a real win.
So we split wins into 2 kinds:
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.