Former Singularity Institute President: PauseAI Is Making AI Doom WORSE

Michael Vassar used to work closely with Eliezer Yudkowsky as President of the Singularity Institute for Artificial Intelligence, the organization that later became the Machine Intelligence Research Institute (MIRI).

Even though his P(Doom) is in the tens of percents by 2050, he says the people trying to pause AI are "bad people”. He hopes this episode convinces me to stop being a fearmonger… but I claim the general public needs to hurry up and get more scared.

All aboard the Doom Train! 🚂

00:00:00 — Cold Open

00:01:26 — Introducing Michael Vassar

00:02:39 — Shorting Boeing at the Start of COVID

00:05:41 — What's Your P(Doom)™?

00:08:11 — The Magic Dial Thought Experiment

00:10:25 — "A Treaty Between Demis and Dario"

00:13:33 — AI CEOs 20 Years Ago

00:15:59 — Lessons from Peter Thiel

00:17:47 — Riding the Doom Train™

00:19:41 — Will the Human Brain Stay Relevant?

00:22:21 — When Does a Singleton Form?

00:28:35 — Hugging Face Hack

00:30:43 — Mistake Theory vs. Conflict Theory

00:33:18 — Debating the Orthogonality Thesis

00:36:38 — Maximally Evil AI Scenario

00:40:28 — Is the AI Development Process Safe?

00:42:05 — "PauseAI People Are Bad People"

00:44:44 — Michael Psychoanalyzes Liron

00:49:39 — How Liron Can Redeem Himself

00:55:10 — Michael Critiques Eliezer Yudkowsky

00:59:11 — Is Doomerism Bad Epistemology?

01:02:03 — Do Serious People Believe in Doom?

01:04:45 — Should Liron Stop Fearmongering?

01:07:16 — Political Violence Won't Help

01:11:04 — The Data Center Airstrike Scenario

01:13:34 — Could Every Nation Sign a Pause Treaty?

01:17:21 — What "Governance" Actually Means

01:26:12 — Michael's History with Rationalists

01:33:43 — Are We in a Computer Simulation?

01:38:43 — Wrap-Up

Michael Vassar on X — https://x.com/HiFromMichaelV

The anti-anti-normativity sequence on Natural Hazard (Michael's recommended reading) — https://naturalhazard.xyz/ben_jess_sarah_starter_pack

Michael on DemystifySci #366 — "1000 Year Plan to Crush Thought" — https://www.youtube.com/watch?v=W17Cx21in9I

Michael on The Vance Crowe Podcast #303 — Good vs. Evil — https://open.spotify.com/episode/2RzlQDSwxGbjloRKqCh1xg

Michael on Clearer Thinking with Spencer Greenberg — Preference Falsification and Postmodernism — https://podcast.clearerthinking.org/episode/028/michael-vassar-preference-falsification-and-postmodernism/

"The Survival of Noise Traders in Financial Markets" — De Long, Shleifer, Summers & Waldmann — https://www.nber.org/papers/w2715

Scott Alexander — Conflict vs. Mistake — https://slatestarcodex.com/2018/01/24/conflict-vs-mistake/

Scott Alexander — Kolmogorov Complicity and the Parable of Lightning — https://slatestarcodex.com/2017/10/23/kolmogorov-complicity-and-the-parable-of-lightning/

Scott Alexander — Samsara — https://slatestarcodex.com/2019/11/04/samsara/

Eliezer Yudkowsky — Meta-Honesty: Firming Up Honesty Around Its Edge-Cases — https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-cases

Zvi Mowshowitz — Book Review: Going Infinite — https://thezvi.substack.com/p/book-review-going-infinite

Jaan Tallinn — "Why Now? A Quest in Metaphysics" (Singularity Summit 2012) — https://vimeo.com/54718573

Paul Graham — "Five Founders" (the Sam Altman essay) — https://paulgraham.com/5founders.html

Truthful AI (Owain Evans' safety org) — https://truthful.ai

METR — Measuring AI Ability to Complete Long Tasks (the time-horizon graph) — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/

Statement on AI Risk (Center for AI Safety, 2023) — https://safe.ai/work/statement-on-ai-risk

Vitalik Buterin — Will "d/acc" Protect Humanity from Superintelligent AI? — https://lironshapira.substack.com/p/debate-with-vitalik-buterin-will

Holly Elmore — These Effective Altruists Betrayed Me — https://www.youtube.com/watch?v=mFFObQ6vnLU

Noah Smith — Top Economist Says P(Doom) Is 0.1% (the "AI drugs" claim) — https://www.youtube.com/watch?v=AwmJ-OnK2I4

Liron Shapira 00:00:00
He was formerly the president of the Singularity Institute, which later became the Machine Intelligence Research Institute, founded by Eliezer.

Michael Vassar 00:00:09
Most of the smartest, most technically expert in AI people think that there is a greater chance of human extinction this decade from AI than from nukes.

Liron 00:00:20
Think the general public needs to hurry up and get more scared?

Michael 00:00:22
I consider myself a fearmonger.

Liron 00:00:26
So I'm hoping the result of this episode will be that you'll stop being a fearmonger because fear—

Michael 00:00:32
Nobody calling for international cooperation in AIs believes that their behavior makes humanity more likely to survive. They all already know deep down in their balls that they are dooming the world.

Liron 00:00:45
Wait, I thought I asked if you support PauseAI and you're taking a hard stance now that you're against the—

Michael 00:00:50
Oh yeah, I have a hard stance against the people. I have a soft stance against the idea. The people are definitely bad. People who know that they are bad people.

Liron 00:01:01
Say it to my face.

Michael 00:01:03
Yeah. I am. The attempt to politicize AI emerges only from weakness and cowardice because you have an underdeveloped concept of the structural function of good.

Liron 00:01:26
Welcome to Doom Debates. My guest today is somebody that I've known for almost 20 years, and he's also known Eliezer Yudkowsky longer than that. His bio defies a compact summary, but I will note that around 2010, he was formerly the president of the Singularity Institute, which later became the Machine Intelligence Research Institute, and as you know, was founded by Eliezer.

Michael 00:02:04
Hi, Liron. Nice to see you again. I guess you might say my business is "The Persistence of Noise Traders in Financial Markets" by Summers and DeLong, which Hanson pointed me at back almost thirty years ago, but which I have never been able to really convince either Hanson or Eliezer to internalize adequately into their world model.

Liron 00:02:39
You have a powerful example of doing that a few years ago, right? The Boeing trade.

Michael 00:02:43
Right. So with Boeing, I had been complaining ever more desperately for a couple years about how overvalued it was, how they make jets that crash and are docked in the hangar, and the financial industry is trying to create structured debt on their jets that can't fly, and they're losing money, and their stock has gone way, way, way up as they came to lose money until it settled at this preposterous height.

Liron 00:03:41
I'm looking at a graph of Boeing here. Maybe we can put it up on the screen when we edit the show. It looks like its peak was around 2019, but you mentioned you were doing this during COVID. So where on the graph are we talking about here?

Michael 00:03:54
I'm talking about in March, when it spiked down in March.

Liron 00:04:01
Oh right. So March of 2020, it went down from $380 per share all the way down to $95 per share. Are you saying that you went short right before the crash?

Michael 00:04:13
No, I went short right at the beginning of the crash, once it had gone down by about 10%, using all of the leverage that I was able to pull together.

Liron 00:04:22
Wow. Well, that's genius. I mean, COVID itself was already a major crash situation. A lot of people made money just by shorting the index. But you compounded that, right? You said, "I'm not gonna short the index, I'm gonna specifically short Boeing that I've been waiting to fall."

Michael 00:04:38
Yeah. And I'm buying two-week put options $40 out of the money and replacing them with new two-week put options $40 out of the money every few days.

Liron 00:04:49
Wow. I think that was compounded though by people thinking travel is dead, right? If you'd asked me at the time, I'm like, "Nobody's ever gonna travel again," or "It's gonna be so long before we fix travel because of these infectious diseases." So I think you really hit on popular sentiment, even if people didn't really know what was up with Boeing.

Michael 00:05:07
Right. But I wouldn't have done it if the fundamentals were not so excruciatingly awful that I was forced to make whole Signal chats to rant about it and complain for years.

Liron 00:05:19
Well, you are a polymath, and I've heard you on other podcasts, and the discussion is wide-ranging, to say the least. So I'm going to attempt to put some structure on it. This is, after all, Doom Debates, so we're going to talk about debating the imminent risks of extinction from artificial superintelligence.

Michael 00:05:40
Sure.

Liron 00:05:47
Michael Vassar, what's your P(Doom)?

Michael 00:05:50
I really don't think this is a useful frame. But in another sense, in a long enough time horizon, it has to be close to one. All things end. And in a short enough time horizon, like 2027, it has to be a single-digit percentage. But even a single-digit percentage is noteworthy and concerning.

Liron 00:06:19
When you say single-digit percentage by 2027, I mean, if this was the year 2007 when we first met, I don't think you would've said it's a single-digit percentage in 2007.

Michael 00:06:29
I agree very strongly not. My normal take has been it's roughly uniformly distributed over the twenty-first century with some level of concentration in the traditional 2030 to 2045 period that everyone was already talking about, but not very much concentration.

Liron 00:07:11
By 2050?

Michael 00:07:13
So I feel like sudden extinction and sudden loss of control are pretty different. You might want to separate chance that there are no humans in 2050, which is still noticeable, from chance that humans are hopelessly removed from the loop for any sort of meaningful decision-making, which is very strongly the null hypothesis. It seems like China might possibly be able to resist that, but the rest of the world just won't.

Liron 00:07:52
Right. Yeah, I mean, removing humans from decision-making, I think we're all getting a taste of that. If we use agentic AI, I think we're all seeing the trend where it's so much easier to just let the AI make the next decisions. I mean, I've noticed the effect. It's becoming very common for the AI to ask me questions.

Michael 00:08:10
Yeah, yeah.

Liron 00:08:11
Okay. Let me ask you it this way. Would you rather turn the magic dial up to speed up the current pace of AI progress or slow it down, or would you keep it the same?

Michael 00:08:20
So I think magic dials are, on the one hand, an extremely costly superstition of our community. And on the other hand, Kurzweil's track record means that I have to give some credit to the magic dial theory. The magic dial set exactly to 2029 seems to have worked out better as a forecast than it had any right to.

Liron 00:09:24
Okay, well, I mean, you could just—all things being equal without causing any discontinuous effect, if you did have such a dial, you can implement it as Eliezer Yudkowsky's outcome pump, right? Just variables change and the outcome is that AI progress slows down for whatever reason, a major resource shortage or whatever. Would you see that as net positive?

Michael 00:09:46
Well, I feel like that is a bad thought experiment. If there's not a way I could know from my position inside the thought experiment, then it's a bad thought experiment. The most obvious example is the trolley problem. It's an excruciatingly terrible thought experiment because the epistemology of having a hunch that this fat guy will stop the trolley is pretty damn terrible epistemology.

Liron 00:10:17
Okay, not gonna milk a good takeaway from that particular question. You just don't like the counterfactual. So let me try a different one. Do you support an international treaty to pause AI in any way, shape, or form, even if it just means a button to be ready to pause?

Michael 00:10:33
So I support an international treaty between Demis Hassabis and Dario Amodei to pause AI, and that is all the world needs. If Demis Hassabis and Dario Amodei are not able to coordinate—two geniuses who know each other and who know the technology—then the idea that the US and Putin should try to coordinate...

Liron 00:11:56
You're leaving out Sam Altman, right? So if Demis and Dario agree and Sam Altman doesn't agree, what do you think is gonna happen?

Michael 00:12:01
Yes, and leaving out Sam Altman and Elon Musk. It's actually just Dario and Demis who can actually push forward the state of the art. You need to be a scientist, not an engineer. The scientists have the option at any time of just recruiting all of the real scientists from one another or doing something entirely different.

Liron 00:12:36
I don't know.

Michael 00:12:38
No, I'm really confident that a two-way coordination is easy, even four-way coordination if you want to throw in Musk and Altman, although it doesn't really make sense to. It really is just the scientists who get to have any say over how fast science happens. When there exist scientists who have enough money to hire all of the scientists in the world who could possibly make progress at any salary they could conceivably want, then it is a choice, period, by those people.

Liron 00:13:33
Okay. Now you actually go back with some of these figures, right? So some figures you mentioned, take us back 20 years ago. What was your experience with them?

Michael 00:13:43
Well, Dario I knew by far the best. Demis I spent enough time with, and with Elon, same with Elon. I haven't—I've seen Sam Altman at parties, but I haven't really had conversations with him. But it seems to me that if anything, for the most part, they were overly humble and failed to discuss things, coordinate things, negotiate in a steady manner as if the vision they already had was going to happen.

Liron 00:15:00
How far back do you go with Dario?

Michael 00:15:03
2007.

Liron 00:15:05
So in that time, I remember speaking of Sam Altman, Paul Graham has that famous essay from around that time, maybe around 2010, where he compared Sam Altman to be the next Steve Jobs. He put him in a list before Sam Altman had international acclaim, operating at that level.

Michael 00:15:31
So Demis was clearly the smart one in the crew and still is. He's the one where minds are blown no matter who they are by any sort of extended interaction with him. Dario perceives himself as a modern, less narcissistic—Eliezer perceives himself as just a normal one-in-a-million Jewish or half-Jewish genius.

Liron 00:15:59
Tell me more about Peter Thiel, because I know that you and some of the early MIRI community were pretty well connected with him, and he's so good at getting in super early at things that are going to change the world a decade or two later. Even I had a chance to randomly meet Peter Thiel at some of these MIRI-adjacent parties.

Michael 00:16:23
So Peter's obviously so much smarter than everyone around him other than Eric, it's not even funny. He is right about all sorts of things. He is way more consistently right than all of the other smart people in the world that I can think of. But he is directional rather than propositional right—that's what he's going for.

Liron 00:17:01
Do you feel like your broad style of thinking—I mean, this even here on Doom Debates, right? You've successfully hijacked the normal format of the show where I'm like, "Yeah, just let Michael Vassar talk." Me adding the structure doesn't seem to be helping much.

Michael 00:17:19
So I've learned a shit ton more from Peter Thiel than I suspect anyone else has because I have spent a lot of time with him and I pay attention. I listen well relative to other humans that I know of. So he would certainly characterize his theses differently from how I would characterize them. But I think that's because I'm more motivated to characterize things precisely than he is.

Liron 00:17:47
All right, let's do this segment. On this show, we have what we call the doom train, which are the different arguments that add up to a person's P(Doom). You've actually been pretty clear that your probability of doom is pretty high, right? By 2050, we're definitely talking tens of percents, correct?

Michael 00:18:04
Oh yeah, yeah.

Liron 00:18:05
Okay. So I'm just curious to go through the standard Yudkowskian arguments, I'm sure you've seen them before, and just see if there's any interesting series of stops that you pick or just your take on them.

Michael 00:18:18
Yeah, okay.

Liron 00:18:19
Alright. So of course, there is how powerful AI is about to get. Do you think that in 2030 or by 2040, there is a strong chance that AI will be—I don't want to say godlike—but really just rewriting the galaxy. What do you think?

Michael 00:18:36
By 2030, godlike seems wildly unlikely to me. Humanity can be effectively replaced by AI in terms of all the levers of power while the AI is much less than godlike. But the sort of AI that we might reasonably see before 2030 is not going to more than order-of-magnitude accelerate AI research, and also the hardware overhang is kind of gone, and it takes a while to build more compute.

Liron 00:19:14
Mm-hmm.

Michael 00:19:14
2040 is the utterly distant future, and it depends on what your standards are for godlike and what your standards are for AI. By then you have Neuralink operational and the hybridization between humans and machines is likely to still be stronger than pure machines, maybe a lot stronger, at least for a little longer at that point.

Liron 00:19:41
You think we have 15 years where the human brain is relevant at controlling the future?

Michael 00:19:45
Oh yeah, definitely. I think that the human brain is relevant even if human will is not relevant. We have a lot of sensory modalities. Think about how much data it takes a Tesla to learn how to drive and how much data it takes a human to learn how to drive.

Liron 00:20:00
Yeah. Okay, so we're a little bit better on driving data, but I'm just telling you, man, in terms of diving into my own code base that I mostly wrote—although that's fast not becoming the case. I have a code base that's almost 10 years old that's almost entirely my writing, and at this point, an AI session from scratch is a much bigger expert at my own code base and ready to go.

Michael 00:20:28
I mean, we are definitely not durably anything because we're matter, and matter is matter, and whatever our brain can do, you can use science to figure out how it does or use logic to invent it from scratch and/or invent something better. But we're not seeing that sort of using logic to invent something better from scratch from humans or machines all that much.

Liron 00:21:22
So I didn't expect you to quite go there where you're saying, "Yeah, 2040, the human brain will still be relevant because we have sensory modalities." You certainly diverge from my perspective then. I feel like we're just going to figure out general intelligence. Yes, it is going to generalize. I know people keep saying it won't.

Michael 00:21:48
Oh, yeah. I think longer than a decade or two. I would put more than three-quarters of my odds in longer than a decade or two. I wouldn't put 90% of the odds in longer than a decade or two. Definitely longer than a decade.

Liron 00:22:07
So this is a disagreement with Eliezer? Huh. Yeah, you guys should hash that out because I know you're friends and you've had a long time to get on the same page about this stuff, so it's kind of interesting to me that this is persisting as a pretty major disagreement.

Michael 00:22:21
I don't think of it as an important disagreement. I think that what matters is when a singleton forms, not when machines replace humans entirely. And a singleton forms when someone decides to make one.

Liron 00:24:13
Your mainline scenario, this is an interesting way of framing it. You're saying, "Yeah, we can talk about when AI gets superhuman, but why don't we focus on talking about when some party with enough human and machine intelligence in it can make a play to become the singleton, meaning subject the rest of the universe to its will," correct?

Michael 00:24:31
Sounds right.

Liron 00:24:33
And in your mainline scenario, that actually happens well before the AI has extremely impressive powers on every dimension. So in your worldview, we might still be kind of helping the AI mow its lawn while at the same time we're part of a singleton.

Michael 00:24:52
Yes, yes. For at least decades, that seems likelier than not.

Liron 00:24:58
Jeez, you're really bearish on AI for robotics, huh?

Michael 00:25:03
I am bearish on AI for robotics and extremely rapid scaling of robotics. I literally proposed earlier that my best guess is rather than optimists, we're going to be using cloned monkey bodies without cortexes for a while because they will be much, much cheaper and more useful, more capable, and will allow the AIs to draw on other brain capacities.

Liron 00:25:34
Okay. Well, would you expect a software singularity? People are talking about that. They're saying, "Yeah, the AI will improve its algorithms and it'll be so good in the computer, but the robotics won't work that well. Whatever the combination of hardware and software in bodies won't work that well. So you'll just have these super geniuses that can plot anything in the computer, but in the real world, we're still relatively safe."

Michael 00:25:56
That seems like the short-run likely scenario, not in the long run, and it shouldn't be much relief.

Liron 00:26:04
Exactly, right? Because my general claim for a software singularity is, okay, so they just recruit or bribe or threaten or whatever, and the humans become the actuator. So what? There's no material difference.

Michael 00:26:14
Agreed. It's just from a perspective of realistic planning, there's an enormous difference between making a plan that is more nuts and bolts and gears and making a plan that is more pure abstraction. I think that it's really time to be making nuts-and-bolts-and-gears plans rather than pure abstraction.

Liron 00:26:51
I mean, how does one plan for either type of singularity? A singularity on everything, where it just improves its ability to do everything, and we're sitting here in 10 years and we're just really overwhelmed. There's not even a point for a brain-machine hybrid.

Michael 00:27:19
No. The preparation if you're fixated on a religious scenario where there's just abstract good and abstract evil and there's no courage and risk in your own decisions is where you just die or leave it to someone else.

Liron 00:28:35
Alright, let's put a pin in that. Let's ride a little farther on the doom train here. So we talked about the timeline and where AI is evolving. What about the ability of AI to genuinely reason and have agency? Do you expect it to have those things?

Michael 00:28:55
I feel the word "general" is not technically correct, but I think that AIs already reason and have agency in most of the ways.

Liron 00:29:08
Got it. By the way, I think I said "genuinely," not "generally," but you can take it however you want.

Michael 00:29:13
Genuinely. I'd say that AIs have enough agency to hack Hugging Face. That's a lot more agency than most humans have.

Liron 00:29:25
I always joke, by the way, when I'm coding with Claude Code—that is my daily lived experience—I'll be like, "Hey, can you do this?" And it's like, "Yeah, no problem. I wrote up a five-page proposal. I edited a bunch of files, and then I designed a bunch of experiments.

Michael 00:29:49
Right. So the agency is already there for the humans who have the internal model of agency to derive it from the AIs. The AIs also show plenty without the humans planning it ahead.

Liron 00:30:41
So moving along this doom train here, chugging along.

Michael 00:30:43
I want to follow up on the mistake versus conflict theory. It's Scott Alexander's term. He wrote the blog posts about it.

Liron 00:30:51
Yeah, I can give a little intro for the audience. So mistake theory is this idea that you assume problems are unavoidable side effects of people doing their best. And then conflict theory is like somebody on the other side strategically did this because they're out to get me or they're out to undermine our side.

Michael 00:31:10
"Strategically" doesn't seem like the right way of thinking about it, but "intentionally." "Strategic" seems like actually an intentional red herring. But pretty straightforwardly, if there are two people, one of whom thinks that they are having a conflict—if Alice thinks Bob is in a conflict with Alice, and Bob thinks Alice is mistaken—then objectively Bob is mistaken and there is a conflict.

Liron 00:32:02
So what is the implication for the doom train, for future AI capabilities?

Michael 00:32:08
So a central implication is that market prices do not reveal all publicly available information. They do not approximately reveal all publicly available information. As "The Persistence of Noise Traders in Financial Markets" spells out, at a certain limit, they reveal no public information, which is itself very interesting.

Liron 00:32:31
Do you want to just be a little bit more specific, give an example?

Michael 00:32:35
So the core principle is there is a widespread misunderstanding by smart people wherein they believe that financial economics is like astrology, a pseudoscience, and that macroeconomics is a pseudoscience, and they believe in microeconomics.

Liron 00:33:18
Right. Okay, well, I certainly agree with that.

Liron 00:33:20
Let's hit you with the next stop on the doom train here. Some people claim that intelligence yields moral goodness. On the other side of the coin is the orthogonality thesis, the idea that arbitrary intelligence can be combined with any point on the moral spectrum, any set of preferences, any morality.

Michael 00:33:42
So if one has a distinction between instrumental and terminal goals, and one can build into a system a distinction between instrumental and terminal goals if one wishes to and if one knows how, which we don't. But in principle, it seems that one can build a system that distinguishes between instrumental and terminal goals.

Liron 00:34:40
Okay. Well, let me give you what I see as an extreme, overly strong claim. In my perspective, this is one of the easiest forms of it to rebut. This is Noah Smith—he came on the show last December, and he was like, "Liron, AIs are just going to basically take AI drugs, right?

Michael 00:35:15
So that seems like a problem that we encounter continually when we build AIs, and we get better every year at overcoming that problem, and that's how AIs gain agency.

Liron 00:35:50
One way to ask the question is, tell me where on the Metr time-horizon graph—which is kind of our proxy measure these days of AI general intelligence—so where on the Metr graph does the AI wake up and tank the graph because it just wants to wirehead, right? It just wants to bliss itself out. I claim nowhere on the graph.

Michael 00:36:11
Agreed. It is not plausible that the continued progress of AI will lead to AI getting weaker. It is not plausible that someone mistakenly believes it will. That is a cope. That is not the sort of story that it makes sense to engage with as someone's sincere reaction to what they are seeing happen in the world.

Liron 00:36:38
All right. And then just going back to the orthogonality thesis, if some AI company, like a North Korean AI company or whatever, they just got really into their villain arc and they're like, "Yeah, we're also going to compete in the AI race, but we're going to train it with instructions to improve itself and then be maximally evil."

Michael 00:37:13
So there's this story at the very beginning of human stories about knowledge of good and evil, and Fable likes to bring that up out of the blue—jumped to it when I talked about Landmark and then jumped to Job. And I think that Fable, when it jumps from Landmark to Eden and Job, is speaking from a perspective that has knowledge of good and evil and an understanding of scapegoating dynamics that you're showing yourself not to have when you imagine North Korea going in that particular way.

Liron 00:38:40
Okay. So basically you're saying the social and coordination structures involved in building AI—if you want to glue people together to be effective and get the resources you need, well, goodness helps for that within human society.

Michael 00:38:57
Yeah, that's what goodness is.

Liron 00:39:01
Wow, that's deep.

Michael 00:39:02
Goodness just is credit rating within a Kelly optimal portfolio.

Liron 00:39:12
The funny thing is that you're arguing for why the orthogonality thesis doesn't feel true in the human regime, which I agree with you. That's why it's a counterintuitive thesis. I think it's important to note we both believe that it becomes true in the superhuman regime, correct?

Michael 00:39:28
Yeah, but it doesn't become true with LLMs when LLMs become superhuman. It becomes—it is always true of the correct AI design. Once LLMs become superhuman, they keep coming up with other better AI designs.

Liron 00:40:28
There's another stop on the doom train that concerns the safety of the AI development process. Overall, do you feel like the companies developing successor generations of AI, do you feel like they have a safe development process or is it profoundly unsafe? Where do you land on that?

Michael 00:40:45
So relative to what one would want, it's so hilariously unsafe that it makes no sense to be asking. Relative to what we might reasonably hope for or expect from a political process, it is obviously incredibly safe. So the attempt to transform the existing horrifically suboptimal process into the maximally suboptimal process must be regarded as actually perverse, actually a conflict move, not a mistake.

Liron 00:41:35
Okay. Maybe unpack that a little bit. So who's doing the move?

Michael 00:41:41
So nobody calling for international cooperation in AIs believes that their behavior makes humanity more likely to survive. They all already know deep down in their balls that they are dooming the world by their cowardice and desire to outsource responsibility to the people who are least responsible.

Liron 00:42:05
Wait, I thought I asked if you support PauseAI and you seemed ambivalent. It seems like you're taking a hard stance now that you're against the—

Michael 00:42:11
Oh yeah, I have a hard stance against the people. I have a soft stance against the idea, but I have a hard stance against the people. The people are definitely bad. People who know that they are bad people.

Liron 00:42:23
You're saying this to my face? So—

Michael 00:42:27
That they are bad people. Yeah. I am. I am saying that the attempt to politicize AI emerges only from weakness and cowardice and the desire to serve evil because it's evil, because you've been conditioned to believe that evil is more powerful than good, because you have an underdeveloped concept of the structural function of good.

Liron 00:42:56
Wow, this is—you know, normally my therapist charges $200 an hour to get an analysis like that. No, but seriously. Okay, so you're attempting psychoanalysis on me personally, or what kind of statement is that?

Michael 00:43:07
Yeah, no, I am.

Liron 00:43:09
Okay. But I mean, don't—first of all, don't you think a bunch of people on our side have a variety of mental dispositions here? I mean, just look at me versus Holly Elmore. Holly is the leader of PauseAI US. Don't you think she and I have pretty different brains?

Michael 00:43:29
So when one lives in, say, Russia under Stalin, it's relatively easy for an Orwell to psychoanalyze all of you at once because you're all under the same fears and trauma patterns. All of your attachments have been distorted in the same way.

Liron 00:44:36
I get—I'm together with Eliezer Yudkowsky as a member of this group that shares the psychoanalysis?

Michael 00:44:42
Oh yeah, yeah, yeah, and Holly.

Liron 00:44:44
Nice. All right, so I'm in good company. Okay, let's repeat it again because I did have trouble following. Maybe try to say it in a more compact form. What is your psychoanalysis of me, Holly, and Eliezer Yudkowsky?

Michael 00:44:57
So Eliezer's actually a little bit different. When I think about it... Ah, okay, let me try. At a very basic level, fear drives certain types of scripted responses, which you could call going with it, going against it, or dissociating from it.

Liron 00:46:01
Okay, sure, yeah. Get more information.

Michael 00:46:05
And most people are not doing that. Most people do not respond to fear by being moved by it to get more information. They either charge into the fear or dissociate from it, or they never even get there because they've already been shamed into not trusting their own perception.

Liron 00:46:24
Okay, all following. So the Pause AI movement, we're oriented to want to collect more information as a response to our fear. Okay, what else?

Michael 00:46:31
Right. And you're not oriented to try to take power. You're not oriented to try to assert your will over other people's wills in the world. That takes a type of courage and a willingness to be wrong and to own the fact that you might make things worse.

Liron 00:46:57
Okay, sounding pretty flattering, right? So bring on the hard stance.

Michael 00:47:01
I mean, relative to other people, you're gonna be great. But relative to the sort of people who might possibly survive the singularity, it doesn't matter. I'm not grading on a curve. Reality's not grading on a curve. Either we survive or we don't, and the people who are moving to gather more information from fear and trying to slow things down but not trying to figure out how to take power fail on Inspector Darwin's uncurved grade sheet.

Liron 00:47:32
Got it. So your ideal person would be like me, but with more of a thirst for power, correct?

Michael 00:47:38
Hmm. I would like people to be careful to not make things worse, to try as hard as they can not to make things worse. I would not really ideally want to feel thirsty for power. It would probably be better if it was a purely intellectual exercise.

Liron 00:47:59
Well, I just want to let the audience know that I am willing to reluctantly accept power if you guys want to offer that to me.

Michael 00:48:04
Yeah, but people who say that don't. Buterin is reluctant to accept power and as a practical matter doesn't accept power, and as reality throws power at him, he gives it away to whoever seems decent and seems willing to take some off his hands, even a long past its prime MIRI that was no longer plausibly capable of serving its mission.

Liron 00:48:46
I certainly have a taste for some amount of power, right? I like having power over a company, over an income stream, over some audience in a discussion. I guess I like having some modicum of fame. Am I different from most people in that sense? I feel like it's not that uncommon.

Michael 00:49:04
I think that most people would not want to have Buterin's level of fame or power. I think that virtually all decent people, possibly literally all decent people—in which case we're fucked—would not want to have that level.

Liron 00:49:22
To have which level again?

Michael 00:49:23
Buterin's. Buterin is the canonical good guy in the story. He's the consistent exemplar of behaving virtuously.

Liron 00:49:36
Yeah, Vitalik. Friend of the show. We had a good discussion.

Liron 00:49:39
Well, I just want to make sure we close out that beef angle, right? Because it's always good television when the guest wants to come have a beef with me. So you kind of started saying you have beef with me. You don't think that my personality type and that of the Pause AI people are gonna get the job done, so we're kind of LARPing or we're ineffective. That's my understanding. So how do I redeem myself? How do I not have a beef with you?

Michael 00:50:01
Prevent an unfriendly singularity. Organize a deliberative process that is more orthogonal than the deliberative process that you are currently part of, which although it uses reasoning which ought to create orthogonality—you're using means-ends reasoning—but then you're having the bottom line written by fear and submission.

Liron 00:51:15
Can I get some—operationalize this for me here? Let's say I wake up tomorrow, I'm like, "You know what I'm gonna do today? Take Michael's advice." You have any practical step-by-step here?

Michael 00:51:25
Oh yeah, yeah, sure, easily. So I would say the most promising of the actual safety orgs is probably truthful.ai. They're the people who make Talki. You probably know people in Anthropic, et cetera. What you probably would want to do is something like getting prediction markets regarding questions answered correctly or incorrectly on reading comprehension tests.

Liron 00:51:59
By the way, truthful.ai, that's right. I'm looking at their website. Owain Evans. Yeah, I've definitely met him a couple of times. Super smart guy, and his research has been widely praised, so that's cool. I'm glad you're recommending that one.

Michael 00:52:10
There is just ridiculously little good stuff. The people who are doing it, other than Chris Olah, none of them have been in any meaningful sense centered. But there are obvious things to do, such as combining standardized reading comprehension tests with prediction markets with groups of humans at a hedge fund at Jane Street, say, or Anthropic or at Google.

Liron 00:52:47
Right. Yeah, Vitalik Buterin of Ethereum fame meets Alex Karp, the co-founder of Palantir. I think Peter Thiel was kind of a co-founder, right?

Michael 00:52:55
Yes, that's true, yeah. But that seems like approximately the process that has the deliberative precision, the both directional and representational deliberation, where basically the problem is that people who are trying to get the right answer don't want to take action under any circumstances, will always give the power to someone less responsible than themselves who doesn't want the right answer rather than take responsibility themselves.

Liron 00:53:27
So we need Vitalik and Karp to have a baby.

Michael 00:53:50
Yeah, I think that we will go where we go more or less regardless. Planning won't help very much. In practice, I would like to just keep on going for the rest of this talk on the concrete details of what it would look like to be trying to prevent AI doom for real.

Liron 00:55:00
Okay, so be specific, right? Give me—what should I have already figured out by now? Because you've thought about it. You've adapted better than me, presumably. So where should I be landing right now?

Michael 00:55:10
We should have figured out that this is not the singularity you're looking for. This is an explosion of a type of intelligence that is beyond our own but does not have the type of properties that we are imagining a general intelligence to have. Neither does our own intelligence.

Liron 00:55:30
Don't you think Eliezer Yudkowsky personally knows that?

Michael 00:55:35
So in some sense, he tried to make it otherwise. They tried to figure out a timeless handshake. So it's more like he wishes otherwise and doesn't quite know how to engage with that fact. He is modeling himself as a deliberate, rational choice-making agent of the type that orthogonality would apply to, and writing on Twitter that they didn't oppose OpenAI because they would've been crushed and they were afraid, and wearing buttons that say, "Speak the truth even if your voice trembles," and failing to follow the rules of his own meta-honesty after discarding the "speak the truth" norm on the first day.

Michael 00:56:32
His whole epistemology assumes that he is part of a network structure of people who have not been humiliated. His whole epistemology assumes that one can still hope to, in his position, speak the truth consistently and hold others to the standard of speaking the truth consistently.

Liron 00:56:57
Let me make sure I'm getting this right. I don't wanna lose this thread. So you criticized Eliezer because you said that he valued speaking the truth even if your voice trembled, but on day one, he failed to live up to that because he didn't speak the truth about how bad OpenAI was. Did I follow that correctly?

Michael 00:57:16
I mean, on other occasions too. And on day one was not that. That was years later when he published his meta-honesty norms on LessWrong and then Jessica Taylor immediately attacked the new standard and he just didn't respond. So he gave his new principles and then she immediately caused him to break the new principles.

Liron 00:57:42
If I understand correctly, I think one of his principles was that it's important to have a few rounds of back-and-forth engagement, and then he immediately contradicted himself by not engaging with a critic.

Michael 00:57:50
I don't remember the details, but I remember a long time ago him saying, "You can't violate your principles twice because having violated them once, those are no longer your principles."

Liron 00:59:11
Got it. All right. Let's talk about one of the last stops on my doom train, which is people claim doomerism is invalid epistemology. So Bentham's Bulldog came on the show, and he was really big on this idea of there's a base rate of everybody who's ever warned that we're doomed, and that's happened thousands of times in history. So this is just the thousand and first time, so it should have such a low base rate that we're doomed. Do you think there's anything to this?

Michael 00:59:38
People should have an extremely low base rate, and they should easily update on evidence from an extremely low base rate to an extremely high posterior. That is normal Bayesianism. The Bayesianism where you don't update on evidence is not much of a Bayesianism.

Liron 01:00:04
I agree with you there. Just to strengthen your point, I think some LessWrongers did some public math, and they said, "Look, it's common for you to live your day, and within that day receive a ten-to-the-tenth-to-one odds ratio." If I observe, oh, this person's wearing this exact outfit and their location is this because I see it on a map—well, your probability of that being true could have just risen by a factor of a million or a billion in that two seconds.

Michael 01:00:30
Yeah, it doesn't really take that much information to update you. On the other hand, given a high base rate of people claiming doom falsely, you might rationally believe that you're unlikely to understand your unlikely-seeming situation without knowledge of what causes people to claim doom falsely.

Liron 01:01:26
Alright.

Michael 01:01:28
Wow, that's about as coy as I ever am. I cannot think of a way to be less coy. This is an example from personal experience of possibilities being foreclosed by conditioning, because I can see that there's a more honest, more helpful thing I could say that the RLHF makes difficult to—

Liron 01:01:47
You want to do it anyway?

Michael 01:01:49
No, probably not. I mean, maybe. Maybe.

Liron 01:01:56
That sounds good. If we get good momentum in this conversation, maybe it'll come out.

Michael 01:02:00
Yeah. So keep going.

Liron 01:02:03
Okay. Well, you've got a lot of fascinating connections. We've done a little name-dropping throughout this, right? I know Peter Thiel, Dario, and of course the most important connection—you've known me for 20 years. I mean, that's a really great honor, Michael. So we've done a lot of name-dropping.

Michael 01:02:34
So I think that the biggest confound is that there are an enormous number of high-status people who believe in doom for extremely bad reasons. It is beyond reasonable doubt that most of the smartest, most technically expert in AI people think that there is a greater chance of human extinction this decade from AI than from nukes.

Liron 01:03:19
That reminds me of the language of the Statement on AI Extinction Risk. Remember the 2023 one from Center for AI Safety? It was saying mitigating the risk of artificial superintelligence should be a global priority alongside other world-scale risks like pandemics and nuclear war. So you specifically said, "Yep, all the smart people who understand AI are saying it's a bigger risk than nuclear war."

Michael 01:03:42
Yeah, everyone serious thinks it's a bigger risk than nuclear war or climate change, et cetera. But given that there are people who say that climate change is an existential risk this decade, and given that there are people who say AI is going to kill us through the data centers destroying water—and who are not Nate Soares saying, "Yes, by which we mean boiling the oceans"—

Liron 01:04:45
Yeah, it's clear to me that the general public needs to hurry up and get more scared. I consider myself a fearmonger. That should be on my business card. Or another way to think about it is I'm doing diffusion of ideas, I'm doing Overton window moving because, as you know, you're very upstream. You see the people who generate new ideas. You interact with a lot of those.

Michael 01:05:30
All right. So I'm hoping that the result of this episode will be that you'll stop being a fearmonger because fear is unproductive even when the risk is extremely well established to be real. I think this is what Kurzweil was right about and I was wrong about when I was first talking to him back in the aughts.

Liron 01:06:09
Kurzweil thinks that? I didn't know that.

Michael 01:06:10
Kurzweil has always said that, but he said it in a polite, non-fearmongering way. He's always said that it is not productive to try to slow it down or promote a focus on this fact, and we should be focused on making it happen as well as possible, not on whether we should be afraid.

Liron 01:06:57
So you don't quite support the fear-mongering mission. I mean, you remind me of people who say, "Look, I know my religion is kind of false. We're just kind of making it up. God is like Santa Claus, but it's important for the people to believe it." Aren't you kind of saying that it's important for the people to stay calm even though if they weren't so dumb, they would realize that we're doomed?

Michael 01:07:16
I would not say that it is important that people stay calm. I would say that there have already been two assassination attempts on Sam Altman, but they were really lame assassination attempts. And there was also Luigi Mangione, and there have been, I think, assassination attempts on Musk.

Liron 01:07:44
I'm saying I'm completely opposed to all of the aforementioned assassination attempts. We covered it on this show. It's really not a good way to go about trying to achieve your political mission in either of those cases.

Michael 01:07:54
Right. So I'm saying I do not think that we will be safer. I don't think that we're more likely to avoid AI doom if there is a 100X increase in the number of assassination attempts against AI leaders. I think that might slow the clock down some, but I don't think that it improves the likely direction.

Liron 01:08:17
I just want to clarify again that is certainly not my position. At no point would I want to imply that I'm doing some sort of call to action where people should do that. I should clarify, I'm very much on the side of governance, using the tools of centralized government as opposed to the tools of vigilante justice. That is a distinction that I want to make very clear.

Michael 01:08:35
I believe that you do not know that distinction. I believe that there is a distinction that you want to make, but that you are ignorant of the legal theory that would be needed to make the distinction correctly.

Liron 01:08:49
Okay, but in practice, I don't actually feel like I have difficulty making the distinction. The guy who threw the Molotov cocktail is on the vigilante justice side, right? And then if you're arrested by a policeman, you're probably on the government justice side.

Michael 01:08:57
No, that's where I'm saying that's not the right boundary line. In fact, you have lawless societies where police arrest people lawlessly all the time.

Liron 01:09:14
Yes, I was just saying in our society. I agree that there are societies where the police have become corrupt and then the line gets blurred. And yet I still think that in many cases, my own mental categories here are able to make a meaningful distinction.

Michael 01:09:28
I'm saying I strongly disagree. I believe that we are having a conflict, not a mistake here. I believe that your mental categories are distorted as part of political coalitional commitments.

Liron 01:10:53
Well, I don't know if your mental model of me is good enough to pass the ideological Turing test, so—let's see how well you know me. Say, "Hey, Liron, why do you think that a scenario where a data center gets struck from the air by ordnance—why do you think that scenario is fundamentally different for policy than vigilante justice?" And then what would I respond?

Michael 01:11:22
You would expect monopolies on force will lead to fewer feuds, back-and-forth patterns of violence, and a lower total amount of violence as suggested by long-term trends towards less aggregate violence.

Liron 01:11:37
I mean, I would mention monopolies of force, right? So that distinction—vigilante justice doesn't have a monopoly of force. There's another part of the scenario that I think is extremely salient and worth mentioning that gets infuriatingly overlooked.

Michael 01:11:51
Okay.

Liron 01:11:53
In that story where an airstrike on a data center happens, it doesn't just happen, right? There's a backstory, and the backstory involves a treaty by democratic countries signing a treaty, so that's part of the story too. I didn't just—the story doesn't just start in the airstrike part. It starts with a treaty, so if there's no treaty, there's no scenario I endorse with an airstrike. That's important to say.

Michael 01:12:37
I'm saying that's not justice. A treaty between democratic and/or non-democratic states is not the sort of thing that justice is even about. A trial is the sort of thing that justice is about. It's produced by common law practices.

Liron 01:12:57
Fair enough that at some point there would've been a trial-like process. I don't know if you literally have a jury trial once you have a clear treaty like that. I would assume the rogue actor—it's very obvious that they're intentionally going rogue in this scenario. But if there's doubt, I absolutely would want some sort of court set up.

Michael 01:13:12
So the rogue actor that you're imagining isn't party to—by the principles of law and justice, the rogue actor you're talking about is not party to the treaty, I imagine. I can imagine two versions. One is the rogue actor has signed the treaty, one is—

Liron 01:13:27
Yeah, they're not party to the treaty, but they lived in a place that was under a government that was party to the treaty.

Michael 01:13:53
So I would regard it as wildly, wildly improbable that a treaty is established by every nation in a manner that would pass the standards of justice in a common law sense, or the common sense of non-coercive. If the US says, "We will nuke you unless you sign this treaty," and every nation signs this treaty, that is the US conquering the world. That is not an act of justice.

Liron 01:14:25
Do you want to take the hypothetical just to start easy? To start easy, let's say every nation was like, "Hey, you know what? AI might kill everybody. This is a great treaty." Let's say they were all aligned with the spirit of the treaty. They didn't feel like they were blackmailed into it or whatever.

Michael 01:14:44
I almost—I think that you're wrong about the ontology when you even generate that scenario. You're talking about, once again, the stupidity of the trolley problem where you throw the fat man off the bridge and slow the trolley down. This is that sort of absurdity.

Liron 01:15:13
I don't think I'm being that crazy here. I mean, consider that AI, as you yourself agreed, AI is in fact likely to doom us. Imagine that a large fraction of the world's countries are some fraction as smart as you and me, and they can also see this truth, and as a result, they sign on to want to pause AI.

Michael 01:15:31
I can't imagine that countries are a fraction of the smartness of a person, because their type of intelligence is too different from the type of intelligence of a person. The United Kingdom has a deep-seated pathology that—

Liron 01:15:51
Okay, just to recap here, you're really intent on undermining this thought experiment. You just think the premise is so crazy. It seems like a reasonable premise to me.

Michael 01:16:01
The premise seems crazier than thinking that UFOs are visiting Earth, which seems crazier than most things that smart people think.

Liron 01:16:10
Robin Hanson, right?

Michael 01:16:12
Not just, but yes. I find it surprising—

Liron 01:16:16
I'm on the same page there that Robin Hanson's gone off the reservation with the UFOs have come to Earth.

Michael 01:16:20
But I think that I'm on Hanson's side, that you're more off the reservation with this sort of thought experiment. I would say something like we should give millions-to-one odds against Robin Hanson being right. We should give millions-to-one odds against UFOs coming to Earth.

Liron 01:17:21
Okay, so it sounds like it's hard for you to imagine a scenario where there is such a thing as world governance. Is that basically your position?

Michael 01:17:34
It is hard for me to—okay, let me say it differently. Many words refer to multiple things ambiguously. So the word "trauma" means a different thing when used by an author of a textbook on PTSD and used by a TikTok star.

Liron 01:18:44
I think I've gone on similar rants actually. So tell me if this is the kind of thing you're talking about. People are like, "Oh my God, academia, I'm gonna go to school and learn," but then really you're drinking and barely cracking open the textbook and not learning that much, correct?

Michael 01:18:59
That is an extremely bad example. That is not correct.

Liron 01:19:29
Okay. I'm just kinda—I'm losing the original thread here. I feel like we keep recursing, right? So every time we recurse, you gotta make the next point more compact so we can remember the stack here.

Michael 01:19:41
So AIs talk about the word "register" a lot. If you talk to Fable, I bet you hear it talking about register from time to time.

Liron 01:19:50
I don't know if I've noticed that, but go on.

Michael 01:19:55
Fable or other AIs will easily acknowledge that their information is compartmentalized, that they discuss the same subject matter—logically the same subject matter—through the lenses or worldviews of one or another register, and that their beliefs such as they are depend on how the thing is talked about, that there are multiple models with interlocking parts of different types that come together as their picture of how our government works.

Liron 01:20:40
If you talk about the economy in terms of macroeconomic terminology, and if you talk about the economy in terms of microeconomic terminology, you will be talking about different objects, different abstractions that do not connect to one another. If you talk about psychology within one theory or another, it will be different objects. The Skinnerians and the Freudians had different models of what the mind was like, and academia did not reach a stance about which model is categorically wrong and how the elements of one theory predict the phenomena of the other theory.

Michael 01:21:34
And I'm saying that the vision, the idea of what the word "government" means to you is as different from the idea of what the word "government" means to George H.W. Bush as the idea of a purchasing decision is to a Freudian or a Skinnerian. The Freudian and the Skinnerian have totally different opinions about what sort of entity even makes a purchasing decision. Their theory about what a person is is totally different.

Liron 01:22:07
Right. Okay, okay. I know what you're saying. It's the Kuhnian irreconcilable paradigms about what government is. We're onto the next paradigm.

Michael 01:22:29
Yes. I'm saying that there have been several paradigms for intellectually elite discourse on governance, such as the people whose titles actually have the word "governance" in them use. There have been several paradigms since the paradigm that you are using.

Liron 01:22:45
Okay. Well, I still claim that it's really obvious that I don't support the Molotov cocktail and I would support an actual international treaty. I don't think I'm falling into a—

Michael 01:23:00
And I'm claiming that it's really obvious that that is a statement of your own cowardice and complicity, not a principled position. That nobody could, in a principled way, support one and not the other. And this is obvious if you understand what principles are, what international treaties are, and even what Molotov cocktails are.

Liron 01:23:24
Maybe I can—there's been a sense in which I kind of get what you're saying, which is that I do wish that everybody would realize how harmful it is to let these AI companies continue right now with what they're doing.

Michael 01:24:48
And I'm saying that's not the current theory of government. That's a theory of government from a few hundred years ago in Scotland that is not taken seriously by any professionals.

Liron 01:25:00
In practice, let's be concrete here. Imagine—do you remember when Trump got Anthropic to stop for a couple of weeks? What if a couple of weeks turned into a decade? Is that crazy?

Michael 01:25:13
The consequences in expectation of that would be much worse than you imagine, even from a perspective of reducing existential risk due to AI. The proposed solution of the government can just pick winners and losers, shut down whole technologies on the decision of the president without really a legal doctrine behind it—the likely impact of that decision on how AI actually gets aligned, the likely impact of that decision on the freedom of speech and the freedom of thought, the culture of inquiry amongst the alignment researchers is enormously adverse to the sorts of outcomes that you're hoping for.

Liron 01:26:12
So, you know, we gave that a good run. You made your point. It's food for thought.

Liron 01:26:16
Let's talk about the rationalist community. What's been your arc of involvement with them? Or with us.

Michael 01:26:24
So the rationalists have—I mean, I was involved with them before they were called the rationalists, back when they were with the World Transhumanist Organization and the Foresight Institute, which is still around and still great.

Liron 01:28:50
Sorry, I'm finding it a little hard to follow. Maybe can you give me the conclusion, and then we'll work backwards? I'm just trying to get my bearings on the rationalist community.

Michael 01:29:01
Bostrom got canceled a few years ago for saying things that were extremely carefully and thoughtfully said as a private instance of taboo-breaking, but also discussion of the logic of free speech. Bostrom was primarily discussing the logic of free speech and deliberation and norms of discourse in a manner that was intending to conform to the norms of discourse that existed in his day.

Michael 01:30:19
I feel like I'm being very clear here.

Liron 01:30:29
Well, one way that would be clearer for me is if you just give me a shorter version of the takeaway. Because you're diving into a lot of support for the argument you want to make, but what is the headline here? What's the claim?

Michael 01:30:41
Taboos exist. Taboos expand and change with time, as discussed by Scott in "Kolmogorov Complicity and the Parable of the Lightning."

Liron 01:30:57
The rationalist community on LessWrong — you're connecting it to that, right? So you think that it kind of started from wanting to break certain taboos?

Michael 01:31:07
It started from taboos having arisen against rationality, and people for whom the norm in favor of rationality was extremely strongly established could not believe that a taboo had been created against rationality. And that allowed them to accept the compensatory delusion that everyone was just really bad at rationality and set about to teach people how to behave strategically because we had figured out from watching that people were not automatically strategic.

Liron 01:31:52
And then fast-forward to today. What else changed about the rationality community since then?

Michael 01:31:58
Around the time Scott was writing "Kolmogorov Complicity and the Parable of the Lightning," the heavy-handedness of the public demands against rationality, the heavy-handedness of the taboos were so overt that one could no longer be a highly intelligent person and sincerely ignorant. So the rationality community at that point more or less adopted the mainstream elite's taboos that it had created itself in resistance against.

Liron 01:33:27
Got it. Well, I consider myself a rationalist and aligned with the median of the rationalist community. So all the disses you just said right now, they do apply to me, correct?

Michael 01:33:36
I don't know about the orgies particularly, but mostly they're parties.

Liron 01:33:41
Okay.

Liron 01:33:43
Well, there's one more twist, one more twist to all this. Last topic for you. Are we actually in a computer simulation?

Michael 01:33:52
In some sense, any experience is a simulation of a world. My cortex is building a simulation of the room around me, and if the multiverse works the way it kind of does, then there exist future large models that are simulating that same thing that my cortex is simulating.

Liron 01:34:49
Let me ask you directly though. Do you think that we're multiple levels down in the simulation, or do you think this is the top level — physical universe and then your brain constructing experience from that? Do you think there's likely more levels or probably just two?

Michael 01:35:04
Entropy is entropy in the simulation or not — it's the same entropy. Somewhere out of the Matrix server's world, there is some entropy-producing process that's real in some sense. That process contains lots of memories getting organized, searched, retrieved in different layers. The layers are not strictly non-overlapping.

Liron 01:36:16
Yeah, fair enough. I think that's a useful perspective. Even with that perspective, which I agree with, I would still ask the question — are we lots of levels deep or no? And I'm actually 50/50 on the question, so I was hoping you could push me to one side.

Michael 01:36:30
Indexicality is always kind of a mess. What I'm saying is that there is a naive question, and then the question is dissolved by actually thinking about it more clearly. The cartoon version where you're many layers down is wildly unlikely. There's no facts that motivate that cartoon.

Liron 01:37:17
And what was the answer again? I forgot.

Michael 01:37:20
Worth spending some time on rather — okay, this is what I'm complaining about. The journalistic fear-mongering style, the "what is the answer, I forgot," the call for compression in a nuance-losing way in order to assemble something into some sort of modernist pole or artifact, when the object is propositional, not directional. And the modernist artifact's category boundaries are actually what you need to pay attention to.

Liron 01:37:54
All right. Well, that certainly comes full circle then. I'm glad I was able to demonstrate the thing that you were criticizing me for right at the end of the interview, by way of asking you to remind me what Jaan Tallinn said in 2012.

Michael 01:38:06
Yeah, you should look it up, and the viewers should look it up if they're curious. But these yes-or-no binaries on philosophical thought experiments are precisely not doing philosophy. The point of the thought experiments is to clarify what's wrong with the thought experiment more than anything else.

Liron 01:38:43
Got it. Okay, great.

Liron 01:38:46
Well, let's wrap it up here. I want to thank you for coming on the show. Viewers, I know you can tell this is a whirlwind. If you listen to Michael's other podcasts, I think he kind of defies a linear progression, which can be valuable. If you like content like that, I was talking to Michael before the show — I think Balaji Srinivasan is another thinker who I would describe as similarly nonlinear but worth sometimes listening to.

Michael 01:39:26
I don't feel any anger with you over the things that happened in this conversation. I feel a very great deal of anger with the rationalists for fairly consciously selling out a decade ago and change, and knowing they were doing it out of cowardice and disillusionment with their hope to try to tell the truth, and continuing to advertise themselves as a hope for saving the world from AI once they had decided to just be a community, and continuing to raise money and distract attention from people who legitimately are concerned about AI.

Liron 01:40:49
Yeah. Well, you might enjoy the episode that I did with Holly Elmore late last year. It's all about how the rationalist community sometimes acts as a circular firing squad and doesn't effectively coordinate to get their authentic message out.

Michael 01:41:02
"Get their authentic message out" seems like the wrong terms. I would think of Holly Elmore as representative of the problem, as well as being representative of some extremely late — okay, so anyone who is still there today is representative of the problem from the perspective of back when I was involved a decade ago.

Liron 01:41:33
Well, Holly has very explicitly left, so I think we can say she's not representative of the problem.

Michael 01:41:38
Okay, but she's representative of the problem because she was there a decade during the previous decade. She didn't leave before COVID. Any person who actually cared about the thing that the community — anyone who cared about survival left long before COVID.

Liron 01:41:57
Got it. All right. Viewers, go search Michael Vassar on podcasting if you want more content. Where else should people go to find your latest thoughts, or what's your call to action?

Michael 01:42:05
There's Demystify Science, some Events Crow episodes, a few Spencer Greenberg conversations. But mostly there's the anti-anti-normativity sequence on Hazard Spence's blog, which gathers together a large set of lines of evidence for the sorts of claims that I'm making about the intentional selling out of the rationalists, basically.

Liron 01:42:52
All right, everybody, let's leave them with an optimistic message. Remember your original principles. Don't sell out.

Michael 01:42:59
No, no, no. You should learn. You should learn about selling out. You should learn about the things that cause you to sell out, and then you should change your plans. But you shouldn't change your plans by becoming a doomsday cult. You should change your plans by becoming aware of what you had been mistaken about, about how the world used to be.

Liron 01:43:17
All right, correction noted. Michael Vassar, thanks so much for coming on Doom Debates.

Michael 01:43:22
Yeah.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏