Which ML models produce the best quantile estimates?

It’s not a secret that most ML models for quantile regression tend to be too biased towards the mean. You want a 10% quantile? The model’s quantile estimate will tend to be too large. And for large quantiles, the estimate is often too low. For example, I used two xgboost models to estimate 10% and 90% intervals, but ended up with an average coverage way below 80% (more like 65%).

So, what is the best model (class) to use?

The short answer: Second-generation tabular foundation models like TabICL v2 and TabPFN-3.0.

These pre-trained models are capable of outputting the full predictive distribution. Meaning you get the quantiles “for free”. Ignoring that TFM inference is kind of expensive, there is at least no additional cost.

The evidence for TFMs being good at predicting beyond the mean comes from the benchmark ScoringBench (website|code|paper):

• This benchmark compares models based on proper scoring rules and other metrics that evaluate the entire predictive distribution.

• An example of a simpler metric of the benchmark is the coverage of the 5%–95% prediction interval (which should be 90%).

• An example of a proper scoring rule in the benchmark is the Continuous Ranked Probability Score (CRPS).

The following figure shows the ranks of various ML models/algorithms:

But what exactly are the fine-tuned models that lead ScoringBench? Since these PFN-based tabular foundation models like TabICL and TabPFN are “just” neural networks, they can be fine-tuned. And if you want them to become better at predicting such scores, you can specifically fine-tune them for scoring-rule objectives. However, even there, it matters which metrics you fine-tune. From the paper: “TabICLv2 was fine-tuned with the CRPS objective, so it improves on CRPS but not on the log score, consistent with the expectation that fine-tuning shifts a model’s inductive bias toward the optimized scoring rule.”

The differences between tabular foundation models and the other ML algorithms seem large in the chart. But they are only large in terms of median ranks. The actual effect sizes are only negligible to small. However, paired with their often stronger performance, tabular foundation models are a great deal, if you can swallow the higher inference cost.

Even fine-tuning doesn’t guarantee coverage; it at best approximates it. Although it seems that TabPFN and TabICL are already pretty good at non-mean predictions. If you need (marginal) coverage guarantees, you would have to go with something like conformal prediction. If you are interested in that, I have a book for you: Introduction to Conformal Prediction With Python.

Anyways, if you need quantiles or anything beyond mean prediction, give tabular foundation models a try.