Toto-2.0: Learning to Forecast Sparse Time Series Without Training

In the previous article, we looked at Toto-2, Datadog’s latest foundation model on a retail forecasting task.

Sparse and intermittent time series remain a challenging setting for zero-shot forecasting models. Many models exhibit what I call failure modes: predicting large, unexplained spikes where the underlying signal is close to zero.

Toto-1 was designed with this problem in mind. It was trained on observability data, where sparse patterns are common. Toto-2 builds on this approach and introduces several architectural improvements. The most important one is the arcsinh normalization adopted from Chronos-2. Unlike logarithmic scaling, arcsinh remains nearly linear around zero, which helps preserve small differences in sparse signals.

In this article, we put Toto-2 through a more demanding evaluation. We focus on challenging examples from the BOOM dataset, introduced alongside Toto-1, and the leakage-free TIME benchmark.

Let’s get started.

✅ Find the Toto 2.0 notebooks of this article here:

➡️ Boom Dataset - Project 30

➡️ TIME Dataset - Project 31

➡️ Retail Forecasting (Project 29 - Previous article)

The first generation of foundation time series models struggled with sparse and intermittent data. A major reason was the lack of high-quality training data. Today, this is much less of a problem. We will return to this point shortly.

Most modern TSFMs handle sparse series surprisingly well. In my view, this comes down to 2 factors: better pretraining and better architectures.

Recent research suggests that models for structured and semi-structured data do not require internet-scale datasets. Instead, carefully designed synthetic data can cover a large number of patterns and edge cases.

This idea has become increasingly important in foundation models for tabular and time series data. Rather than collecting massive amounts of real-world data, researchers can generate diverse training examples that expose models to situations they may rarely encounter in practice. A good example is the use of Prior-Fitted Networks (PFNs), first introduced in the TabPFN line of research. Toto-2.0 uses PFNs to generate synthetic time series, including sparse and intermittent patterns.

PFNs are more than data generators. They leverage structural causal models (SCMs) to sample complex relationships from a distribution of priors. This allows them to create realistic and highly varied forecasting scenarios, including many edge cases that are difficult to find in real datasets.

Pretraining is only part of the story.

Foundation forecasting models have also gone through several architectural iterations. Benchmarks and ablation studies provide useful guidance, but many design choices still require empirical exploration. As is often the case in deep learning, some improvements become obvious only after extensive experimentation.

One change that stands out in Toto-2.0 is the arcsinh normalization.

Unlike logarithmic scaling, arcsinh is approximately linear around zero. This preserves the small differences that characterize sparse and intermittent time series. At the same time, arcsinh behaves like a logarithm for large values, compressing extreme spikes and reducing scale differences across datasets.

This combination is particularly useful for foundation forecasting models. Small differences remain visible, large outliers become manageable, and the model can learn from datasets during pretraining that span several orders of magnitude.

Adapted from here, I created a mini-benchmark to examine whether a model can handle sparse data in the first place.

Let’s try the smaller Toto variant (4m) first:

The model avoids the spurious spikes we often see on zero-inflated windows. However, it still misses the main peaks and fails to capture the gradual increase in peak magnitude.