Reading · The Oracle by Polymarket

Werner K. Zagrebbi is a board-certified Superforecaster™ who writes about politics and economics at Right Rationalism.

If you take the trip from DC down into that intricate amphibious Northern Virginia region, there could often be seen sauntering across Fairfax a very interesting old man, like a wizened sage—portly—with a glint in his blue eyes and lightness on his feet testifying to a glorious twinkhood lost.

In the words of his colleague Bryan Caplan: “When the typical economist tells me about his latest research, my standard reaction is ‘Eh, maybe.’ Then I forget about it. When Robin Hanson tells me about his latest research, my standard reaction is ‘No way! Impossible!’ Then I think about it for years.”

It could be argued Hanson originated modern prediction markets: If he gets the Nobel Prize (Manifold put his odds at 20%), it’s expected to be for prediction markets.

I came across his work in his capacity as one of the first niche internet microcelebrity bloggers, and I saw in him an intellectual hero worth idolizing. Beginning in High School, I often took the pilgrimage to George Mason’s campus, and faced him over games of Dungeons and Dragons. This month, I went back with a recorder.

Here’s our conversation, condensed and edited for clarity.

Well, you know, prediction markets just boomed in the last few years, and all of a sudden lots of people want to talk to me. That’ll probably go away when it becomes less of a novelty. But at the moment, it’s a novelty.

Once you realize you want the people who value the information to pay for it, the obvious first place you’d go is companies. They need the information, they’ve got money, and their decisions are valuable. That’s what people tried to do. In the aughts there was a lot of effort to get companies to do these markets. And what we learned is this: most managers present themselves to the world as scientific decision makers. They’ve got a spreadsheet, they’re trying to fill in the numbers so they can make the best decision for the company. Under that model of who they are, they should want these markets. But that’s not who they are. They’re actually politicians. They have an alliance and they’re trying to support it. They’re trying to recruit allies to support decisions, and they don’t want some random neutral source weighing in that messes up their ability to control the narrative.

My longer-term history is: I learned the usual points of view in economics as a student, and then I noticed lots of puzzles, things that didn’t make much sense. This is one of them, but it’s only one of many puzzling things about the world. My book The Elephant in the Brain is the realization: what if people are just wrong about their motives? Then you can explain a lot of things. And I think that works here too. Managers’ motives aren’t actually to be the scientific decision maker filling in the spreadsheet.

I have this hypothetical of the autist in the C-suite. You put a really smart person there who understands the company really well, and every time a subject comes up they just spew out lots of relevant, important things about it. They would not be allowed to sit there, because other people can’t rely on them to support their agendas. They would be disruptive. But that’s what a prediction market is. Something that speaks the truth without any sense of who wants to hear it.

The most common scenario with consulting is that somebody wants to make a decision but doesn’t have enough internal support, so they hire McKinsey to be the outside support for what they’ve already pre-decided to do. McKinsey isn’t actually doing that much original analysis. They’re offering their support. It’s credible in part because there’s only so far they’re willing to go. If you bet wrong that McKinsey will support what you want, you lose out. But that is the main thing they do.

There was this guy Poindexter, of Iran-Contra fame, very famous, who was in the hierarchy above us at DARPA. Some reporters said: how can we find dirt on him? So they looked at projects under him, and they found our project. To get him fired. He wasn’t fired for Iran-Contra or whatever. He didn’t have much influence over the project, but he was nominally in charge of it, so that was the basis.

They made sure to have their press conference when the DARPA PR person was unavailable. It was a Monday morning, and by the next morning the Secretary of Defense said our project was killed. They did not have much time in 24 hours to assess the situation. They killed it on the basis of: we don’t want to be associated with this bad publicity. And they did not check whether the accusations were even correct.

The other people on the project just wanted to hunker down and hide. I was the one person willing to talk to the press, so I went through several months of talking to the press, and I learned a lot about how these sorts of issues play out.

There’s also the power of accusing norm violations. If you’re associated with an accused norm violation and you say, let me get back to you on that, I need to look into it, that looks bad. You’re supposed to have an automatic response. A norm violation is supposed to be so obvious, and your position on it so obvious, that you don’t need time to react. This is of course not true. In reality almost all norm violations have details you might not understand, and we don’t always agree on what the norms are. But we have this norm that we DO agree on the norms and that their application is obvious. That’s why, when you’re accused, you don’t have time to look into things.

It did happen, ten years later. IARPA, a related but different government organization, funded prediction markets, and my team’s pitch was: we’ll do the combinatorics. That was SciCast, roughly a thousand markets, starting with geopolitical events in the East and moving to science and technology. We did it ten years later without much publicity, because Poindexter wasn’t in the hierarchy above us.

For a while, maybe ten years, the CIA had internal prediction markets, and they were useful to them. But they were only allowed to exist because of one rule. The coin of the realm in the US intelligence community is white papers. White papers cite other white papers. What you aspire to do in that world is write white papers that get cited. So the rule was: the markets can’t be mentioned in white papers. They were allowed to exist as long as they didn’t get credit. Individuals could participate, look at the prices, let them influence their opinions. But in the white paper they presented themselves as having thought of all these things themselves.

I don’t know what killed it. My guess is that eventually people higher up heard about it, and when they see a white paper they say: what does the market say? Now it’s getting credit. You can’t let that happen.

Institutional innovation is just more difficult, because you’re threatening people’s status and power. People are much more okay with innovation that happens in a lab: if the lab succeeds, they change the production process of a product, and the whole organization stays the same, except now they’ve got more money. Innovation of institutions, of who gets rewarded and what we believe, is much more threatening. So you should expect it to be harder to get experiments started, harder to get them to last, harder to produce innovation. And you can’t patent institutional structures. It’s a basic fact about the world: we are much more eager for innovations in computation and physical devices than in innovations of social practices. The biggest social innovation of the last few decades was social media, which was presented as a technology. People were okay with introducing it as a technology, and then it had social implications as a side effect.

The world of people I was hanging out with in Silicon Valley in the late 80s. I left in ‘84 to go out to Silicon Valley, and I went away in ‘93 when I went to grad school. In that world there were lots of big ideas for the future that people were throwing around, and I was trying to contribute to that. In retrospect we could say I added prediction markets to that world, and that was successful. Other people were doing other things. Hal Finney, of Bitcoin fame, was one of my friends in that world. He’s my best guess for at least part of Satoshi. Eric Drexler, who many people thought was more likely to be the harbinger of the future, for nanotech. That turned out not to be such a thing yet. There were people trying to make social media at the time. There was cryonics, which I got into eventually.

Share The Oracle by Polymarket

When I first came to the subject, I was imagining: let’s just have legal markets on the questions we all argue about in public policy and politics and the media, and that would be enough. That’s the initial vision, and I think it’s the vision most people start with when they come to the subject. Every few years more people come to the subject, and that’s where they start.

There are three things I think people should eventually come to realize. First, the world has far more topics that matter than the topics that show up in the news. Second, the markets we’ve had so far are mainly a side effect of traders who are there for risk hedging or entertainment reasons. What we want is to switch to a model where the people who want information pay for it. Then we could unleash a much bigger market. And third, decision markets: if we want to make things relevant for decisions, we make the markets right next to a decision.

But even on the first milestone, look at the number of news articles that show up in a year and the number of markets that show up in a year. We’re still less than one percent. We have a long way to go even to reach that milestone, and that milestone is still much smaller than the bigger long-term vision.

That’s the key open question. Years ago I wrote a blog post listing 21 ways firms seem inefficient. You might think that for each of them competition will eventually force the more efficient thing. But it’s an open question. I still hope, and there’s a chance that’s worth pursuing even if it’s not obvious it’ll happen. If you know the literature on innovation, the world just has far less innovation than would be possible. We’re proud that we innovate much faster than people did three centuries ago, and that’s true. But compared to what seems possible, we’re still way below.

That’s where I got my inspiration. Other people had this idea of the World Wide Web, and I thought that was cool. I went to Silicon Valley in part to work with them on that idea. They had a vision: they thought making criticism easier to find would help conversation. And I had doubts. I thought, is that really going to work? It probably has helped, but it’s not nirvana. That’s when I asked, well, what else could we do? And that’s when I started to think about prediction markets. In that context of big, grand things we could do with new technology to change the world.

The LessWrong community has a lot in common with the Xanadu community. Similar sorts of people, a generation apart, trying to do similar sorts of things. The Xanadu people thought of themselves as thinking rationally about which technologies would be beneficial and how, and as better able than other people to see possibilities. That’s what the rationalists have seen themselves as, too. I don’t know that one caused the other, but whatever forces caused one probably caused the other.

There were other people associated with Xanadu who were also in that community. Still around, still having influence.

People inspired by Eliezer are still predicting FOOM [explosive acceleration in AI ability]. Soon, to happen in a few years. They say it hasn’t happened yet, but it’s about to happen. So to settle this bet, we just wait a few years and see if it happens. Ask me again in ten years. Then I’m going to look pretty right. They will not have seen the FOOM happen in the next ten years. Of course, they may say it’s still coming then.

On ems: if AGI shows up in the next five years, then it shows up before ems, most likely. There are people working on ems right now. But obviously a lot more attention has gone to the usual kind of AI. There are many orders of magnitude more money in AGI than in ems.

I see Eliezer’s argument as: it’s possible for a very basic algorithmic change to have this huge impact, where it allows the machine in the basement to take over the world on the weekend. And if that’s true, we need to prepare by giving all the AIs good values through that transition. My critique was that the sudden explosion is not very plausible. It’ll happen slower, we’ll have more chances to react, and it will be less concentrated, spread across the whole world economy, not just one machine in the basement. He has been saying lately: see, Robin, you were wrong, because look how algorithms matter. And I say: we do have algorithms that matter, but they matter when paired with lots of data. The big ventures take pretty widely known algorithms and combine them with specialized implementations and data to be the leaders in these industries. The difference between the leaders is often data advantages, hardware advantages, market position advantages. Not just algorithm advantages. Algorithms matter, but not in the way that a machine takes over the world in a weekend matters.

There have been forecasting efforts over the last decade where groups tried to get people to forecast different aspects of AI predictions, and then asked whether that helped resolve differences. It apparently did not. When people break the argument into pieces and argue over the pieces, that hasn’t reduced the difference between the groups. And in a way it did for other topics. There was one study with several topics, and on the other topics, breaking things down and engaging the details brought the groups together. Not on AI. They had subject matter experts and general smart superforecasters engage on the details, and on the other topics they came closer to each other. On AI, the superforecasters said: no, we don’t believe it. Sorry. So the AI debate is somewhat different from these other debates. The technical details just don’t make a big difference

Our world is too much structured by status as the default organizing principle of institutions. If you don’t have a prediction market and there’s some topic we’re uncertain about, the usual thing is: who’s highest status here? What do they say? That’s how we handle hard questions. I would rather replace that. I want status to matter less as the way we run parts of our world.

In my world as an intellectual, I hear people getting status for doing impressive things. And I go: no, I want to give status to people who made an impact, who actually made a difference, not just did something impressive. A lot of academic work is impressive but not very useful. People tell me I’m smart, and I go: I wasn’t trying to be smart, I was trying to accomplish something. If you’re impressed with what I’ve accomplished, I’ll own that. I deserve the status when I did something that mattered. But I don’t want to hand out status for potential, or impressiveness, or way with words, or the way he holds himself and commands authority. I want actual accomplishment. I’ve met Elon Musk. He’s not that impressive as a person. For real. But damn it, he deserves status, for what he’s done in the world. I don’t know who deserves more. So hand out status for actually doing stuff, not for how someone holds their head when they’re standing next to you, or talks fast, or uses a big vocabulary. In a prediction market, people who trade and make money because they moved the price in the right direction: that’s a real accomplishment. Give them status.

The one who’s made money.

OG

Comments