Forecasting and Fantasy

The Great British Bake Off (GBBO), or the Great British Baking Show here in the US, is a delight to watch. I love a competition show, and GBBO gives the genre an uplifting spin. Further, the show is chock-full of baking inspiration. After Dylan’s gochujang rolls earned a coveted Hollywood Handshake last year, I spiced up my favorite (American) biscuit recipe at my family’s Thanksgiving.

Each season, my friends and I participate in a GBBO Fantasy League, complete with punishments for the loser: everyone votes on one challenge for the (un)lucky participant to recreate. I’ve avoided this fate so far, albeit narrowly. I came closest when I tried to scrape past seasons’ data from Wikipedia. The data cleaning process was tedious, and I did little better than propagating last week’s performance into the upcoming week.

This year, I’m planning to delegate. In practice, this means I’ll end up spending far more time on the problem, at least in the short run.

Humans have been working on statistical forecasting problems for some time. Computers are quite helpful in this effort, dating back (at least) to ENIAC. We’ve been working on judgmental forecasting problems for even longer. I’d argue that much of our decisionmaking involves implicit, judgmental forecasts. I don’t possess a sharp characterization of the differences between statistical and judgmental forecasting because I don’t think there is one. Judgmental forecasting problems are statistical forecasting problems: a base rate calculation is a simple statistical analysis that is quite useful in many judgmental forecasting questions. Statistical forecasting problems are judgmental forecasting problems: modelers must make judgments with respect to modeling approaches. Some judgmental forecasting problems might progressively become statistical forecasting problems as we learn to tame unstructured data.

I think that selecting the winners in GBBO requires a fair amount of judgment. I tried a statistical approach when scraping data from Wikipedia, but I could not generate a rich feature space. This year, I wanted to use a language model to generate features from transcripts,1 but I can’t access timely transcript data. So, the problem remains judgmental in nature.

While I won’t use a language model to turn this problem into a statistical one, I don’t need to give up on the LLM hype train. All aboard.

I spend quite a lot of time thinking about judgmental forecasting. In fact, it’s how I pay my bills. Judgmental forecasting is time-intensive, and I have a lot of respect for the (super)forecasters in our studies at FRI. It is not easy to source a large quantity of predictions for our various forecasting programs. In fact, it is one of our chief challenges. We could think about delegating the problem of forecasting to LLMs, and there are some exciting new efforts to do exactly this. Lightning Rod Labs hosts Foresight, and Mantic just emerged from stealth mode.

So, I posed the problem of picking season winners and episode 2 results to Foresight. I plan to do the same for the rest of the season. I’ll refrain from giving my thoughts on the platform until I get more comfortable using it.

Judgmental forecasting is challenging. These tools are new, and I suspect they will get better. I am happy that talented people are building them!

I have been excited about the extending frontier of forecasting tools for the past few months. I’ve gathered my preliminary thoughts on what features I’d like to see in these tools. I’ll start with this “delegation” approach, in which we hand over forecasting questions to tools that drive independently to an answer.

I want to interrogate the arguments of an automated forecaster. Consequently, I first and foremost need visibility into the reasoning for a given forecast. Then, I want to probe this reasoning. How does the forecast on the ultimate question change if I modify some assumptions or introduce a new consideration into the problem? In other words, I think automated forecasting should be a multi-turn problem.

We could also think about removing the human from the interrogation stage. Can you compel an automated forecaster to conduct a premortem? Or, could you task another agent to play the role of the devil’s advocate?

I would like to equip an automated forecaster with tools specific to the problem of judgmental forecasting. For example, we could provide an automated forecaster with an agent devoted to tracking down base rates for a number of reference classes. I would also like to be able to compel the automated forecaster to use this tool.

An aggregation of multiple forecasters’ reports often outperforms its constituent parts. This phenomenon appears to generalize to LLM forecasters. This wisdom-of-the-crowd process should be especially suitable to LLMs. Indeed, Lightning Rod Labs already seems to have this feature. Nevertheless, it remains unclear how to optimally seed the different ensemble members.

But, I think we can take this a step further. The output from an automated forecaster is typically a single point in an idea-forecast space. However, an ensemble gives us many points in this space. It’s possible that this richer output, consisting of competing perspectives, would be more palatable to high-stakes decisionmakers, provided the debate can be condensed into a digestible format. However, I suspect the lack of accountability poses a larger barrier to adoption than do clarity and concision.

Every innovation I’ve described above can be slotted into this delegation model. However, it’s not clear to me that we should be delegating the process of judgmental forecasting to AI systems. Would these systems multiply our ability to model the future, or would they instead atrophy this important skill?

I’m also interested in a model of “collaboration.” We might interact with AI systems to help us improve the calibration, discrimination, and clarity of our expectations. Under this model, I could imagine:

• A forecasting coach that points to pitfalls in our reasoning and suggests additional exercises/considerations (I’ve started a demo here, which nudges a user to calculate a base rate and provides some basic hints)

• A suite of forecasting tools that a human (instead of an AI system) dispatches. This point is closely related to the “Specialized Tools” section above. Could I dispatch a web browsing agent to assemble base rate data from a set of reference classes I develop?

We already delegate many of our forecasting tasks, but this effort has been limited largely to statistical forecasting problems. If you can generate a set of relevant features and collect some historical data, you can spin up a statistical approach to a forecasting problem.

However, not all problems are amenable to this treatment. Statistical forecasting problems need some degree of exchangeability between past and future events. That assumption might be quite tenuous in some contexts, especially for rare events. It is worth noting that the suitability of judgmental forecasting approaches to these rare events and/or ones without historical precedent is an open question. If you’re interested in answering that question, you should work with us at FRI.

In a statistical forecasting problem, I care about the quality of the input data and, sometimes, transparency in the algorithm. For, e.g., weather forecasting, I do not mind that the algorithms are opaque to me. They are not opaque to some individuals, and their performance can easily be audited on out-of-sample tasks. It is not clear if any single contributor even understands the whole.

In judgmental forecasting, I also care about the inputs. Which pieces of information are used in the forecast and how influential is each source? Next, the analog of algorithmic transparency is reasoning transparency. I want to understand how a forecaster makes the leap from qualitative to quantitative judgment, be they carbon- or silicon-based.

Note: Photo by GuerrillaBuzz on Unsplash.