Predictions Over What?

I love predictions. I love prediction markets, prediction contests, and expert predictions. I love having AI make predictions as an alignment strategy, and I love using predictions to align AI.

Not everyone shares my love of predictions. Some critics make hateful comments, like “If predictions are so great, why don’t people use them to make decisions?” or “It sure seems like a lot of prediction market volume is degenerate gambling”.

As someone who has previously called prediction as the core of intelligence, I am unfortunately forced to admit that the critics have a point here. Predictions are beautiful in an abstract sense, and may be the key to AI alignment in the future, but as of today aren’t worth much.

Scott Alexander wrote fairly recently about this issue with prediction markets. Despite a huge increase in usage over the last year, that hasn’t led to actionable insights. Some hypotheses he throws out are that the markets don’t have the right questions, the resolution criteria depend on minutiae, and that knowing more accurate probabilities for many ranges doesn’t neatly map to different decisions.

Friend of the blog Daniel Paleka more directly addressed the issue with AI forecasters in a post titled Even superhuman AI forecasters are only as good as your questions.

Consider ACME Hardware, an American company that procures raw materials and other goods from various Asian countries. Their supply chain is complex. They want to predict the future to make better decisions. To be precise, they need to decide which contracts to sign with which suppliers, while minimizing the chance that supply chain disruptions will stop their production.

Forecasting platforms already have questions like “What will the US tariffs on Malaysian goods be in 2026?” and “Will there be a war between China and Taiwan in 2026?”

But ACME Hardware executives don’t care about these questions! They want to know what to do. They want an answer to: “Should we sign the contract with the Malaysian supplier or the Vietnamese supplier?”

For prediction market advocates, the question of why ruthlessly profit-seeking companies don’t use them has been a challenge. Robin Hanson suggests that the reason is that junior employees speaking truthfully creates internal political issues. But even if this is true, it suggests the upside to prediction markets is limited enough that political considerations dominate them everywhere.

Regardless of how good a market or AI is at making predictions about the world, users have to translate those predictions into concrete actions. An advantage of AI over markets is that the user can easily query them for predictions over a personally specified set of outcomes, but even that can be hard to interpret or miss important information.

If we create AI that’s superhuman at forecasting, we can probably use it for something, but much of the value could remain locked behind the relevancy of its predictions. This is a particularly important consideration for proposals like Oracle AI (or Scientist AI, as the cool kids are calling it these days), which even in the best case trade off some capabilities for safety.

So, relevancy is a major known problem for AI predictions. Coefficient Giving included the issue in their recent RFP on AI for Forecasting and Sound Reasoning. What can we do about it? If you know my approach to alignment, you won’t be surprised to hear I propose defining formal incentives for decision relevancy.

The core mechanism for how to do that is based on an approach I first heard suggested in an old blog post from Katja Grace, called How to buy truth from a liar. The idea is that if you don’t know what information to ask an expert about, you can pay them each time you change your mind. You might go back and forth many times, but eventually you get to the point where nothing else they tell you will affect your decision, so you can think of it as a well-informed choice.

Consider a decision between two possible actions you might take. Each has a conditional prediction associated with it, giving the distribution over outcomes if it is taken. There are two possible sets of outcomes you might want to use:

1. The minimal set of outcomes where adding more details would not affect the decision

2. The minimal set of outcomes where the action that would be taken under full information is chosen

With the first, splitting an outcome into different sub-outcomes won’t change the decision maker’s mind, while the second allows that so long as further splits would change it back.

The “buying truth from a liar” setup can be simplified to avoid the indefinite back and forth by using two experts, which is already necessary to elicit honest conditional predictions. The desired result is that in any equilibrium, all experts provide the same set of outcomes, matching either the first or second criteria.

To elicit a set of outcomes of the first type, the mechanism is to have one expert propose a set of outcomes, then give a second expert the chance to refine it by splitting outcomes into suboutcomes. If this refinement does not change the decision maker’s mind, the first expert gets rewarded proportionally to the simplicity of their set of outcomes. Otherwise, the first expert is penalized and the second expert is rewarded.

To elicit a set of outcomes of the second type, use the same setup, but give the first expert a chance to respond. If they change the decision back to what would have been decided after their initial proposal, they get rewarded and the second expert is penalized.

I have proofs showing that each of these mechanisms leads to the desired behavior, and they can be made zero-sum if collusion is a concern. These rely on the experts choosing the outcomes being different than the experts making the predictions, but I am close to proving that the separation is unnecessary.

Theoretical results are one thing, but let’s suppose the mechanism works in practice. What could we do with it?

The most straightforward use is better decision making via AI forecasters. Rather than relying on existing prediction markets, or trying to come up with our own questions to ask AI, we can have the expert AI point out what is relevant.

Among other benefits, this lets us elicit predictions over continuous variables, without needing to throw away information (or create extraneous information) by coming up with our own partition. We can also adapt the mechanism to continuous decisions (such as what rating to give on a 10 point scale) by checking whether more information changes the decision by some set amount, rather than changing it at all.

With multiple decision makers, the outcome sets that would be provided to any of them can be integrated, so that all of them receive any information that would matter to any of them. With predictions over facts made common knowledge, debate over values can be more focused.

The “outcomes” being predicted include the histories that lead to the final states. A relevant prediction can then highlight the sequence of events causing the end result, providing a sort of explanation. This can be useful for explaining even deterministic trajectories, pointing out the relevant components. I see this as related to half of the ELK problem, generating explanations where humans can assess their correctness.

Actions can also be provided alongside a set of outcomes. The decision maker changing their mind can then come from new actions and/or more information about actions. This also comes in handy if the actions under consideration are underspecified.

Besides augmenting human decision making, I’m excited about potential applications of relevant predictions to training AI. In a typical actor-critic setup, the critic head predicts average reward, which is then used to train the actor head. Breaking down that prediction of the average into the relevant outcomes (including both physical results and human approval) opens up new possibilities for how to use them.

A straightforward application is in upgrading MONA. MONA works by training AI on human predictions of average reward, avoiding reward hacking by only reinforcing actions where the human anticipates good results. However, this introduces a performance penalty when humans cannot see how aligned but complex actions will pay off. Instead, using human approval of outcomes predicted by AI can get the safety benefits of MONA without the performance penalty.

With predictions over outcomes available, we can also evaluate actions with metrics other than mean reward. For example, if an action leads to high expected approval, but only due to a very small chance of extremely high approval, we can assign a lower reward to discourage such risky actions. Many possible transformations are available, affecting either the probabilities or the approval scores.

When predicting average reward and comparing it to received reward, there is no way to distinguish between unlikely events occurring and poor assessments of approval for a given event. Predicting a joint distribution over outcomes and approval given outcomes lets assessments of particular outcomes be updated more precisely. This also allows for flagging outcomes where the AI has uncertainty as to the human evaluation, even when those outcomes are not realized, allowing them to be targeted with further training.

Of course, supposing the mechanism works in practice isn’t enough, it has to be actually implemented. I’ll be working on that, and related problems, as my next major project. If you’re interested in working with me, or have relevant thoughts, please reach out!