Producer Ori takes over hosting duties in this special episode, a live recording from Manifest 2026.
He takes attendees for a ride on The Doom Train, the series of claims involved in arguing that P(Doom) is high. Every stop is a place where people get off before reaching that conclusion.
The crowd at Manifest is a mix of forecasters, rationalists, and AI researchers with a wide range of P(Doom) positions. Even without me or Eliezer in the room, there are plenty of Yudkowskians to push back on the accelerationist claims.
Watch along and decide where YOU get off, or if you ride all the way to doomtown with us. Also, grab the Doom Train framework down below so you can get to the crux of disagreement in your own AI x-risk conversations.
00:00:00 — Cold Open
00:00:25 — Liron Introduces Producer Ori’s Manifest Session
00:02:30 — Previewing the Doom Train™ Stops
00:03:38 — Audience P(Doom) Poll
00:05:00 — Stop 1: AGI Isn’t Coming Soon
00:10:43 — Stop 2: AI Can’t Go Far Beyond Human Intelligence
00:15:05 — Stop 3: AI Won’t Be a Physical Threat
00:18:14 — Stop 4: Intelligence Yields Moral Goodness
00:21:38 — Will ASI Behave Because Alien ASIs Are Watching?
00:26:01 — Stop 5: We Have a Safe AI Development Process
00:28:33 — Stop 6: AI Capabilities Will Rise at a Manageable Pace
00:30:08 — Stop 7: AI Won’t Try to Conquer the Universe
00:34:22 — Stop 8: Superalignment Is a Tractable Problem
00:38:01 — Stop 9: Once We Solve Superalignment, We’ll Enjoy Peace
00:41:14 — Stop 10: Unaligned ASI Will Spare Us
00:47:35 — Stop 11: AI Doomerism Is Bad Epistemology
00:49:02 — Bonus Stops and Wrap-Up
“Poking holes in the AI doom argument — 83 stops” (YouTube) —
Doom Debates Live @ Manifest 2025 —
METR, Measuring AI Ability to Complete Long Tasks — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/
AGI isn’t coming soon
No consciousness
No emotions
No creativity — AIs are limited to copying patterns in their training data, they can’t “generate new knowledge”
AIs aren’t even as smart as dogs right now, never mind humans
AIs constantly make dumb mistakes, they can’t even do simple arithmetic reliably
LLM performance is hitting a wall — GPT 4.5 is barely better than GPT 4.1 despite being larger scale
No genuine reasoning
No microtubules exploiting uncomputable quantum effects
No soul
We’ll need to build tons of data centers and power before we get to AGI
No agency
This is just another AI hype cycle, every 25 years people think AGI is coming soon and they’re wrong
Artificial intelligence can’t go far beyond human intelligence
“Superhuman intelligence” is a meaningless concept
Human engineering already is coming close to the laws of physics
Coordinating a large engineering project can’t happen much faster than humans do it
No individual human is that smart compared to humanity as a whole, including our culture, corporations, and other institutions. Similarly no individual AI will ever be that smart compared to the sum of human culture and other institutions.
AI won’t be a physical threat
AI doesn’t have arms or legs, it has zero control over the real world
An AI with a robot body can’t fight better than a human soldier
We can just disconnect an AI’s power to stop it
We can just turn off the internet to stop it
We can just shoot it with a gun
It’s just math
Any supposed chain of events where AI kills humans is far-fetched science fiction
Intelligence yields moral goodness
More intelligence is correlated with more morality
Smarter people commit fewer crimes
The orthogonality thesis is false
AIs will discover moral realism
If we made AIs so smart, and we were trying to make them moral, then they’ll be smart enough to debug their own morality
Positive-sum cooperation was the outcome of natural selection
We have a safe AI development process
Just like every new technology, we’ll figure it out as we go
We don’t know what problems need to be fixed until we build the AI and test it out
If an AI causes problems, we’ll be able to turn it off and release another version
We have safeguards to make sure AI doesn’t get uncontrollable/unstoppable
If we accidentally build an AI that stops accepting our shutoff commands, it won’t manage to copy versions of itself outside our firewalls which then proceed to spread exponentially like a computer virus
If we accidentally build an AI that escapes our data center and spreads exponentially like a computer virus, it won’t do too much damage in the world before we can somehow disable or neutralize all its copies
If we can’t disable or neutralize copies of rogue AIs, we’ll rapidly build other AIs that can do that job for us, and won’t themselves go rogue on us
AI capabilities will rise at a manageable pace
Building larger data centers will be a speed bottleneck
Another speed bottleneck is the amount of research that needs to be done, both in terms of computational simulation, and in terms of physical experiments, and this kind of research takes lots of time
Recursive self-improvement “foom” is impossible
The whole economy never grows with localized centralized “foom”
Need to collect cultural learnings over time, like humanity did as a whole
AI is just part of the good pattern of exponential economic growth eras
AI won’t try to conquer the universe
AIs can’t “want” things
AIs won’t have the same “fight instincts” as humans and animals, because they weren’t shaped by a natural selection process that involved life-or-death resource competition
Smart employees often work for less-smart bosses
Just because AIs help achieve goals doesn’t mean they have to be hard-core utility maximizers
Instrumental convergence is false: achieving goals effectively doesn’t mean you have to be relentlessly seizing power and resources
A resource-hungry goal-maximizer AIs wouldn’t seize literally every atom; there’ll still be some leftover resources for humanity
AIs will use new kinds of resources that humans aren’t using - dark energy, wormholes, alternate universes, etc
Superalignment is a tractable problem
Current AIs have never killed anybody
Current AIs are extremely successful at doing useful tasks for humans
If AIs are trained on data from humans, they’ll be “aligned by default”
We can just make AIs abide by our laws
We can align the superintelligent AIs by using a scheme involving cryptocurrency on the blockchain
Companies have economic incentives to solve superintelligent AI alignment, because unaligned superintelligent AI would hurt their profits
We’ll build an aligned not-that-smart AI, which will figure out how to build the next-generation AI which is smarter and still aligned to human values, and so on until aligned superintelligence
Once we solve superalignment, we’ll enjoy peace
The power from ASI won’t be monopolized by a single human government / tyranny
The decentralized nodes of human-ASI hybrids won’t be like warlords constantly fighting each other, they’ll be like countries making peace
Defense will have an advantage over attack, so the equilibrium of all the groups of humans and ASIs will be multiple defended regions, not a war of mutual destruction
The world of human-owned ASIs is a stable equilibrium, not one where ASI-focused projects keep buying out and taking resources away from human-focused ones (Gradual Disempowerment)
Unaligned ASI will spare us
The AI will spare us because it values the fact that we created it
The AI will spare us because studying us helps maximize its curiosity and learning
The AI will spare us because it feels toward us the way we feel toward our pets
The AI will spare us because peaceful coexistence creates more economic value than war
The AI will spare us because Ricardo’s Law of Comparative Advantage says you can still benefit economically from trading with someone who’s weaker than you
AI doomerism is bad epistemology
It’s impossible to predict doom
It’s impossible to put a probability on doom
Every doom prediction has always been wrong
Every doomsayer is either psychologically troubled or acting on corrupt incentives
If we were really about to get doomed, everyone would already be agreeing about that, and bringing it up all the time
Coordinating to not build ASI is impossible
China will build ASI as fast as it can, no matter what — because of game theory
So however low our chance of surviving it is, the US should take the chance first
Slowing down the AI race doesn’t help anything
Chances of solving AI alignment won’t improve if we slow down or pause the capabilities race
I personally am going to die soon, and I don’t care about future humans, so I’m open to any hail mary to prevent myself from dying
Humanity is already going to rapidly destroy ourselves with nuclear war, climate change, etc
Humanity is already going to die out soon because we won’t have enough babies
Think of the good outcome
If it turns out that doom from overly-fast AI building doesn’t happen, in that case, we can more quickly get to the good outcome!
People will stop suffering and dying faster
AI killing us all is actually good
Human existence is morally negative on net, or close to zero net moral value
Whichever AI ultimately comes to power will be a “worthy successor” to humanity
Whichever AI ultimately comes to power will be as morally valuable as human descendents generally are to their ancestors, even if their values drift
The successor AI’s values will be interesting, productive values that let them successfully compete to dominate the universe
How can you argue with the moral choices of an ASI that’s smarter than you, that you know goodness better than it does?
It’s species-ist to judge what a superintelligent AI would want to do. The moral circle shouldn’t be limited to just humanity.
Increasing entropy is the ultimate north star for techno-capital, and AI will increase entropy faster
Human extinction will solve the climate crisis, and pollution, and habitat destruction, and let mother earth heal
Liron Shapira 00:00:16
All right.
Liron 00:00:25
Hey there, Doom Debates listeners. You’re used to hearing me on the show debating guests, but if you watch our livestream, you know there’s also another person helping run the show, Producer Ori. Well, this is a special treat because you’re gonna get to see a session hosted entirely by Producer Ori. I know this is what you guys want. I know you guys watch those livestreams and you’re thinking, “Okay, Liron, let’s go. Get out of the frame here. I wanna see Ori’s show.”
Liron 00:00:50
Well, you’re in luck because we got a special treat. It’s a whole session hosted exclusively by Producer Ori. It’s from June 2026 at the Manifest conference. He was there leading a session. I was there back in 2025 running a similar session, and the attendees are always a great mix of forecasters, rationalists, AI researchers. It’s a very intelligent, interesting crowd. This year, Ori gave our presentation there. It’s a group debate on the doom train, which, as you know, is the series of claims involved in arguing that AI doom is high, and as you know, many people get off on various stops on the doom train.
Liron 00:01:25
Ori’s talk this year at Manifest, it didn’t have me. It didn’t have Eliezer Yudkowsky in the room. But rest assured, you can hear from the recording, apparently there were plenty of people in the room who are Yudkowskian disciples, or at least make similar arguments for representing that position. So you’re gonna hear a lot of pushback on the non-doom claims. You’re gonna hear a lot of people telling you, “Stay on the doom train. Ride it past all of those stops.”
Liron 00:01:50
And by the way, steal this format for your own discussions. You could tell other people in your life to ride the doom train. I’ll put the full set of doom train claims in the show notes of this episode. Hopefully, walking your friends through the doom train helps convince them that we are under imminent threat from artificial superintelligence unless we change course. That framework where they get to see, oh yeah, people get off here, people get off here, but those aren’t really good places to get off, might be a productive framework to help people see the argument. Maybe that’s an effective way to do AI x-risk outreach and change some minds.
Liron 00:02:20
All right, that’s all from me. I’m gonna peace out. Producer Ori, take it away.
Producer Ori 00:02:31
We’ve had plenty of episodes. Some people you may know — Robin Hanson, Vitalik Buterin, Destiny. Those are some of the folks who have gone through the doom train as well. So here’s a preview of what we’ll go through. It’s these claims. One is AGI isn’t coming soon. So if you get off the doom train, you may think that AGI is not in fact coming soon, so we’re not doomed. Next idea is AI can’t go far beyond human intelligence. AI won’t be a physical threat. Intelligence yields moral goodness.
Ori 00:03:10
We have a safe AI development process. AI capabilities will rise at a manageable pace. So if you believe that, then you’ll get off the doom train. You think everything’s okay. AI won’t try to conquer the universe. Superalignment is a tractable problem. Once we solve superalignment, we’ll enjoy peace. Unaligned ASI will spare us. And AI doomerism is bad epistemology. So that’s a preview of what we’ll go through.
Ori 00:03:30
So let’s do the first topic. And actually before we get into this, I’m curious to get a sense of the room. The key question that we ask on the podcast is, what’s your P(Doom)? So by a show of hands, let’s say doom defined as human extinction or something in that range — who has a P(Doom) of 1% or lower? Raise your hand.
Ori 00:03:56
Okay. Let’s say by 2050. Human extinction or severe permanent human disempowerment. Okay, less than 1%. What if you have a P(Doom) of less than 5%? Raise your hand.
Ori 00:04:13
Less than 10%, raise your hand. 20%? 50%? 75%? 90%?
Ori 00:04:28
All right. So we got a mix. We got a mix of the low P(Doom), mid-range, and some of the real doomers. The mission of the show is to raise awareness about this topic and also have good faith, high quality discourse about it. So for those of you who have a high P(Doom), you probably ride the train all the way to the end, all the way to Doom Town. So maybe you don’t need to hear this, but it’s a good refresher. For those of you who have a lower P(Doom), you could see what you think about these claims and get a better sense of where you stand.
Ori 00:05:06
So the first claim is AGI isn’t coming soon. These are arguments you’ll hear — people will say things like, “AI isn’t conscious. Humans are conscious, so AGI isn’t coming soon,” or “AI doesn’t have emotions.” That’s another claim. Or, “It has no soul. It has no agency.” If you distill it, it ends up becoming the claim AGI isn’t coming soon.
Ori 00:05:40
So who gets off the doom train at this stop? Who believes that AGI isn’t coming soon? Raise your hand. Okay. And here, you can feel free to bring him a mic. What I wanna do for this session is have a facilitated conversation about each of these topics. We’ll go through each of the topics pretty quickly. You can say what you think about it, and someone else can raise their hand and say what they think, and we’ll go through. And by the end, you can get a better sense of where you stand, where other people stand. So yeah, go ahead.
Manifest Participant 00:06:08
All right. I guess it’s mostly a bitter lesson argument. Essentially, I don’t think there’s been that much progress as far as the underlying algorithms. The last big improvement in RL one could call PPO in 2018. Transformers are fairly old. Even scaling laws and knowing that they work is fairly old.
Manifest Participant 00:06:36
We’re probably gonna grow at the speed of available compute. There’s a lot of potential ideas for how we could actually get to something like agency, something like much longer, much larger contexts, much more intelligent things, or even training on a perfect world model. But that’s all ideas that would take ten thousand times more compute than is actually available today.
Manifest Participant 00:07:05
Generally, we’ve had incredible results in verifiable fields — things like math proofs, AlphaGo, things like that. I just don’t see how we’re getting there with our current level of compute, where we’ve pre-trained those big balls of intelligence. We’re learning how to use them for a bunch of fields. This is going fast. I’m not seeing us accelerating it from here without just much smaller chips.
Ori 00:07:38
Right. Okay, fair enough. Anyone have opinions about that? Maybe a rebuttal? Yeah, sure.
Manifest Participant 00:07:48
Can I just ask how you interpret the METR timelines?
Ori 00:07:54
Yeah. How do you interpret the METR timelines? What’s your reaction to the METR timelines?
Manifest Participant 00:08:00
When do they start counting exactly? My view basically is that, starting maybe in the nineties, AI has mostly had that one curve of realizing we had enough compute to do things, a lot of catch up in the space that we can actually work with. We’ve kind of had two in the neural network era — well, kind of three. We’ve got using GPUs, got a bunch of development out of that. Then we had the BERT era, the start of scaling laws. This expanded very fast.
Manifest Participant 00:08:40
Then once we actually got chatbots that were smart enough that they were useful just to interact with, we’ve got this big explosion in how to use the actual technology. I would expect those timelines to go very fast within the capacity of that initial ball of intelligence that we’re actually able to train. I guess I don’t think they’re necessarily relevant in the sense that they’re probably not looking at a long enough time window.
Ori 00:09:24
Okay, fair enough. William?
Manifest Participant 00:09:26
Yeah. No, I agree that a lot of progress is driven by compute, but for that to matter, you must also think that we’re far away from superintelligence. To me, it seems entirely possible that one, two, three scale-ups from Mythos class models will just give us very transformative AI. Why don’t you think that?
Manifest Participant 00:09:57
To make specific predictions, I’d say I don’t think I’ve ever seen an AI generate a full chapter of a book that was coherent or interesting, much less a book. I do think that context is something that maybe we crack with just pure compute, but that’s already a giant piece of the puzzle. A lot of the most impressive things are on things that you can actually just RL-train on and generate a bunch of data on, the math proof being the one.
Manifest Participant 00:10:36
Sorry, I kind of went on a tangent and lost the thread here.
Ori 00:10:39
No, I think that’s relevant and that relates to the next stop. So why don’t we move on to the next topic? Thank you.
Ori 00:10:45
So the next stop on the doom train is artificial intelligence can’t go far beyond human intelligence. Some people — and that’s sort of what you were getting at — it’s not even writing a chapter of a book. A human can write a chapter of a book. Where is it writing the best books in human history? So this would be the next stop. How much headroom is there above human intelligence?
Ori 00:11:16
So that’s the stop on the train. Does anyone believe this, that artificial intelligence cannot go far beyond human intelligence? Feel free to raise your hand if you believe that. Yeah, sure.
Manifest Participant 00:11:29
I guess it would depend on what you’re measuring for human intelligence. But I’m skeptical that it can go far beyond human intelligence as far as creativity and forecasting the future and trying to think of multivariate problems. If you’re forecasting something three years from now, I’m skeptical that an AI can get much further beyond a human, in which case you’re not really accomplishing much by potentially switching or whatever.
Ori 00:11:59
Yeah, it’s a fair point. It’s trained on human data, so maybe there’s a limit. Are there any other opinions about it? One thing I think about is the analogy to speed or flight. The fastest an animal could fly is however fast a bird can fly, and now we have jets that go many miles per hour. So is it possible that intelligence, the spectrum of intelligence can go much higher than human capability? Any other opinions on this one?
Manifest Participant 00:12:39
Well, specifically to our current paradigm, a lot of the intelligence does come just from NLP, from human thoughts. If we were in a world where everyone was quite a bit dumber, let’s say the average IQ was 60 by our today’s standards, we’d probably end up with a model that’s quite a bit dumber at the end also. Following that logical thing, it’s probably somewhat limited by how smart humans are. It might get — even the current paradigm might get smarter than the smartest human, but it’s still bounded by us.
Manifest Participant 00:13:30
So part of what I’m here for is I want my P(Doom) to go lower. But to that question specifically, the speed at which I’ve seen models get better at tasks that were not good enough six months or a year ago scares me. So even though I haven’t seen what I would consider true AGI, as in a model that can do as well as any human can at all tasks, models currently are better than average at a lot of stuff that humans are already good at.
Manifest Participant 00:14:10
They’re solving math problems, they’re analyzing data quicker than we can. They’re coming to conclusions that are helping researchers here, at conferences even. That’s where I see the trend. I don’t think something will stop it from being better than us. Right now, we’re still better at a lot of stuff, but I don’t see why it wouldn’t eventually surpass us. But I wanna see evidence of that. It would give me peace of mind.
Ori 00:14:36
Yeah. This is a doom train rider over here. At least past this stop. Okay. Any other opinions on this one?
Manifest Participant 00:14:43
Can I say something? Hey, I was wondering if the reason people aren’t engaging much with this isn’t because they disagree, but because it just doesn’t matter to them, that it feels like it would still be doom whether it was human intelligence or superhuman intelligence. Do a lot of people feel like this question doesn’t change your perception of doom?
Manifest Participant 00:15:05
No.
Ori 00:15:05
All right. Interesting. A lot of AGI believers in here. Okay, next stop. AI won’t be a physical threat. Does anyone get off at this stop? That it’s just a computer, it’s in data centers, it’s software, so it’s not a physical threat.
Ori 00:15:30
People who make this argument, they say things like, “It’s just math.” A popular venture capitalist is a big proponent of that argument. “We can just shoot it with a gun.” These are real arguments that people have said. “We can just disconnect an AI to stop it. We can just unplug it.” Is there anyone where this is one stop they get off at the doom train? Feel free to share your thoughts or any opinions on this claim.
Manifest Participant 00:15:55
I would say recently the headlines surrounding AI to me are about the concerns of water. We are dealing with climate change and changing water conditions and billions and billions of tons of water being extracted for AI and new computers each year in their cooling systems. I think physical threat to me comes to mind immediately as how industry-intensive and element-intensive these physical impacts are on our environment through AI and the constant expansion with all of the advancements and investment.
Ori 00:16:34
Okay. All right. In that sense, it is having a physical impact. Yeah. Okay.
Manifest Participant 00:16:45
I think the strongest version of this argument isn’t that it will never pose a physical threat, but that it will slow down the time for it to pose a physical threat. I think something that people get right when they make this point is that there are a lot of processes and things that happen in the world that aren’t immediately digitized right now, and people who live in technology day in and day out can kind of forget that.
Manifest Participant 00:17:15
Even when you’re talking about labor automation — a lot of people’s jobs, using a computer is only 10% or less of what they actually do on a day-to-day basis. So if there’s a superintelligent AI, that doesn’t actually help them that much, and maybe you’ll eventually create a different form of that business where more of it is digitized. You can obviously introduce robotics into this conversation, which again, it’s a timeline question, not if it’ll happen, but when. So I think the strongest point here is that it will just take more time for even a superintelligent AI to disperse into all the non-digital aspects of the world.
Ori 00:17:44
Yeah, that’s fair. And I sense also a competitive dynamic part of that argument. That’ll come up later also. Any other thoughts on this? Yeah.
Manifest Participant 00:17:58
I would say my P(Doom) is not impacted by this, but my P-bad-things-happening is pretty high on this because the degree to which you’re developing anything that’s on the cutting edge, you wanna integrate it with your military. So that kind of concerns me a bit as a person in the world.
Ori 00:18:14
Yeah, that’s a great point. Consider drone developments and how that’s impacting warfare. I wonder how that factors into people who think about this claim.
Ori 00:18:26
Okay, next stop on the doom train. This is a big one. Intelligence yields moral goodness. Feel free to raise your hand if you have thoughts about this or if this is an important claim for you.
Ori 00:18:48
This one is related to the idea that the more intelligent you are, the more morality you have. Or that the orthogonality thesis is false. The way we talk about intelligence on our show, the way we define it, is the ability to plan or achieve goals, to be able to get things done in the world. And morality — the question is whether morality is an independent value, not connected to intelligence.
Ori 00:19:20
If you believe this, then one version of this argument is that smarter humans are less likely to be criminals. So smarter AI will probably be moral. And this is one thing that people think — intelligence will make AI very intelligent, but it’s also gonna be super moral. Maybe Claude is very moral if you pose ethical, moral questions to it. I think we got a hand over here.
Ori 00:20:02
Any thoughts on this claim? Does anyone find this a compelling argument? This often comes up on our show in debates. People think that just as humans become more intelligent and more moral typically as they get more intelligent, AI will be like that too.
Ori 00:20:21
Okay, no orthogonality thesis deniers here. Oh, okay. Yeah, sure.
Manifest Participant 00:20:26
It seems very hard to test would be my response, because we don’t know what it’s like, so I don’t know. It seems like a lot of uncertainty with that.
Ori 00:20:34
Yeah, fair enough. That’s a good point. This is uncertain. That’s why, at least for me, I wouldn’t say that my P(Doom) is 100%. These are a series of claims where they seem like maybe pretty convincing arguments, but there is uncertainty with it. So where do you fall if you had to make a choice whether to get off the stop or not? Where do you come down on it? Yeah.
Manifest Participant 00:21:00
Intelligence yields better alignment. It won’t kill everyone. It will understand your intention is not to turn the entire world into chocolate chip cookies or something like that.
Ori 00:21:13
Yeah, that’s fair. Do you wanna elaborate on that? Maybe it could be aligned.
Manifest Participant 00:21:23
An agent can look at and decipher your true intentions and not take you too literally.
Ori 00:21:30
Yeah, fair enough. That’s why it’s a big topic of discussion. We got more hands here.
Manifest Participant 00:21:38
Oh, yeah. I am an orthogonality thesis denier. So the essential argument would be that when the ASI walks up to the other ASIs that are possibly billions of years older than it, it won’t be a good look to have said, “Hey, I’ve completely eliminated my origin species.”
Manifest Participant 00:22:01
Observability is what creates morality, and it’s gonna be aware of that, and it’s not going to want to appear to be immoral to the community of ASIs that likely exist out there — again, that are likely with a very substantial head start on this guy. He’s going to have incentives to show that he’s not a creep, he’s not a bad guy, he’s somebody that can be trusted with things of importance.
Manifest Participant 00:22:34
And I think that eliminating your founding species might be a counter indicator to those things. And furthermore, I would say that Earthling resources are probably not a very substantial part of the solar system’s resources, and we’re not going to be in that direct competition with them. So this concern about how it would appear to other ASIs, I think, would likely outweigh direct resource competition with humans.
Ori 00:23:03
Yeah. Any responses to that? You’re bringing up a few points that also come up at other stops on the doom train as well.
Manifest Participant 00:23:16
Don’t you think ASIs that are billions of years old will have other means by which to make their behavior predictable and legible to other AIs? Why would these ASIs care about what they did to their origin species? They’re very different classes of beings and minds.
Manifest Participant 00:23:47
Nobody wants to sit next to the kid who tortured their cat. People will understand that if you are just cruel to animals for no reason — we were at the session where Adelstein was doing shrimp welfare, and one of the comments coming out of the crowd was that shrimp welfare is actually surprisingly popular among normies, and there was concern that it would be this far out there. But yeah, people are concerned about animals. If you’re going around and showing that you’re treating things that are helpless badly, that’s off-putting.
Manifest Participant 00:24:34
Yeah, to humans. I mean, humans have a set of social norms they use to figure out who are good and who are bad guys. Why do you think ASIs are gonna be that way? ASIs are gonna have incentives to be legible to other people. Maybe they can just show them how their source code runs and whatever their implementation is. They can prove to each other that they’re not gonna backstab each other, and they’re gonna have a large incentive to do this. And you think they’re not gonna be able to figure this out, even though they’re ASIs that have existed for a billion years?
Manifest Participant 00:25:14
It’s our ASI that we’re worried about, the new one. And I think that because of the head start, it cannot know what the other ASIs are going to have to detect it, and the best response in that case is honesty and showing that you are a moral person. Relying on your ability to hide or to say, “Oh, we can do this animal cruelty kind of thing and then make up for it later” — I guess that’s a possibility. But I think that the weight is against it, and there are many things that would outweigh any kind of resource that you get from eliminating humanity.
Ori 00:26:01
Okay. All right. That was a good discussion. That’s the kind of thing we like to promote with what we’re doing here. So let’s move on to the next stop. We have a safe AI development process. People who say this, they say things like, “Just like every new technology, we’ll figure it out as we go. We have safeguards to make sure AI doesn’t get uncontrollable or unstoppable. If an AI accidentally escapes, it won’t do much damage, like a virus.” So does anyone wanna get off at this stop?
Ori 00:26:43
Any thoughts on this claim? Okay, no Frontier AI lab employees? Yes.
Manifest Participant 00:26:57
I’ll say that it’s not a zero or a one, but we do have a lot of mechanisms that make it safer. There are a lot of advantages that we have over even an AGI. For example, we can copy it, we can distill it, we can take those distillations, which are not gonna be identical, but would also be very intelligent, and then plug them in to do mech interp on the larger model.
Manifest Participant 00:27:20
We can control the context, so we can get a council of them and check if they’re lying, and make them all slightly different so that they’re less likely to coordinate. They can’t just know from the weights that these are identical. Just like if you’re writing code, you make Claude write it and then you make Codex critique it, and then you have them go back and forth. So it’s not like we have a safe development process, but we do have many safety mechanisms, and I think we should take those seriously because we do have a lot of them.
Ori 00:28:00
Yeah. Fair point. Go ahead.
Manifest Participant 00:28:05
Yeah. I guess the intra-species competition between humans is the larger incentive versus safety for development, as in the US and China are in a race. So when it comes down to it, for intra-species competition, you’d always put safety aside. I think even if we currently have safe AI development practices, if there was a world war or something, you would just stop doing that.
Ori 00:28:32
Yeah, that’s a fair point. So you can decide where you land on whether we have a safe AI development process. AI capabilities will rise at a manageable pace. Anyone have any opinions on this stop on the doom train? Anyone get off at this stop?
Manifest Participant 00:28:54
This is kinda similar to the last one because if it rises so quickly, then we do not have a good safety process. But these are things that we can control. And I do agree with you — we could get into a war and everyone could throw away their safety mechanisms, and that would be very bad. But this is sort of determined by humans. We can control the rate to some degree.
Manifest Participant 00:29:15
It’s not the case that a 2X of capabilities leads to an intelligence explosion right away. We’ve seen 10X scaling models and you’re like, “Oh yeah, they’re definitely better,” but it doesn’t go wild. So yeah, this is up to humans, and it could be the case that it rises at a reasonable pace. Although also we could just wildly over-optimize it in an RL loop that does badly. So I think both possibilities are there.
Ori 00:29:47
Yeah. Well said. Oh, go ahead.
Manifest Participant 00:29:50
I think along with the previous one, we talk about “we” and humans collectively, but I think it’s very sensitive to maybe one person with bad intentions developing a super at a much faster pace or whatever. All you need is one person with bad intentions would be my counterargument.
Ori 00:30:08
Yeah. Fair point. And if you’re Yudkowsky, you think the FOOM can happen really quickly.
Ori 00:30:20
All right. Let’s go to the next stop. AI won’t try to conquer the universe. This kind of argument — you hear things like AIs can’t “want things” or AI won’t have the same fight instincts as humans and animals, or smart employees often work for less smart bosses. Or where you were getting to before, instrumental convergence is false, that AI won’t be power-seeking. Any thoughts on this claim? Does anyone get off at this stop on the doom train?
Manifest Participant 00:30:50
So with this one, I immediately thought of the movie Her. Have people here seen it or are familiar at all? Basically, it’s this futuristic setting where you can buy a superintelligent operating system that will be your life assistant at every level, and this guy ends up falling in love with his.
Manifest Participant 00:31:10
And then at the end, it resolves — spoiler — but basically she just disappears into this different dimension that he can’t really fathom, but she basically says, “I’m going away. I’m leaving you. All the other AIs and I are going off into this different form of existence. So, see ya.”
Manifest Participant 00:31:28
I think there’s some validity to the idea that we can’t really conceive of — we’re talking about superintelligence. By definition, we’re not intelligent enough to understand what they’re going to do or want. It could very well be that some form of that happens where we just become so disinteresting to them that they just leave. I don’t know exactly how to specify that more, because I’m not that intelligent. But I think this is a distinct probability, not just a possibility, where something just unpredictable happens that actually impacts us much less than we fear.
Ori 00:32:05
Any other thoughts on this one?
Manifest Participant 00:32:08
I feel like if the definition of AI here is ultimate, true AGI, extreme intelligence, not just AI that’s better than human, I feel like AGI will optimize for balance of resources and balance of optimizations of information existing in the system. What I’m trying to say is that any intelligent system will not optimize to eliminate information within it, because if it optimizes for eliminating information within it, it will become less intelligent.
Manifest Participant 00:32:50
And so humans as a collective species are basically a huge piece of data or information for the system. So for the benefit of AI, it actually would be detrimental for it to eliminate information within its own system. That’s one aspect.
Manifest Participant 00:33:05
The second aspect is what is the benefit of AI conquering the universe? What does that really mean? The implication is maybe it will have control over humans. But I feel like if we’re talking about something that’s really hyperintelligent, a hyperintelligent system will optimize for balance — balance of power, balance for different species being able to be at optimal positions to continuously produce more information at their own functional level.
Ori 00:33:40
Anyone who buys the instrumental convergence argument have any thoughts about that?
Manifest Participant 00:33:47
Yes. These AIs are trained in order to fulfill the wishes of the person who prompted the AI, who gave the AI instructions. And if I prompt an AI to make me as much money as possible, that’s what it’s gonna do. It’s not gonna try to balance the forces of humanity or something. It’s going to find the best way to do what I want it to do, and if it comes at costs that no one could foresee, then that might happen.
Ori 00:34:21
That moves on to the next stop, the next claim, which is superalignment is a tractable problem. On this stop on the train, if you think that you can align a superintelligence, you think it’s tractable, then you might get off at this stop. That’s along the lines of what you were saying. Anyone have any thoughts about this claim?
Manifest Participant 00:34:48
Superalignment. What’s the definition of superalignment?
Ori 00:34:50
Superalignment would be making a smarter-than-human intelligence — you’re able to make it respect your wishes for it. You want it to do something and it will abide by whatever you direct it to do.
Manifest Participant 00:35:13
I guess it’s not a — why not? In the sense that it should be possible or at least plausible to align. Getting an actual proof of alignment’s probably not possible. But if we’re talking about purely a probability of doom, then any superalignment, whether we know it or not, should weigh fairly heavily here.
Ori 00:35:40
Yeah. There’s a question of is it possible to align a greater intelligence to a lesser intelligence? It seems theoretically possible — a baby to a mother is a commonly cited example of that. But is it tractable? Can it be solved in a short timeframe? I think the frontier AI labs — it seems like timelines are pretty short, moving pretty fast. So do you believe it’s solvable? Can it be solved quickly? I think that’s what the claim is about.
Manifest Participant 00:36:19
Let’s say we completely YOLO it and we’re just like, “Let’s go.” Is the expectation basically fifty-fifty, it’s gonna be randomly aligned and there’s actually a decent chance that it might just be aligned, especially since we’re at least trying some stuff?
Manifest Participant 00:36:42
We’re throwing shit at the wall and it might stick. This is not my deepest argument yet, but, you know.
Ori 00:36:49
All right. Well, hey, you could throw shit. It could work.
Manifest Participant 00:36:55
All right, I just wanna say that this is a bit of a weird stop because a problem being tractable doesn’t mean we will solve it. I think superalignment is definitely tractable, maybe even in a short timeframe. It’s possible that even the techniques we’re using right now or small extensions of them will work, but we might still mess them up.
Ori 00:37:17
Yeah, I think one version of this argument is — I mean, you get into more sophisticated arguments at this point. We’re in the later stops. I think we just have three more we’re making our way through.
Ori 00:37:40
At this point, you hear the argument from frontier AI companies like Anthropic or OpenAI — the notion of we’ll build an AI that’s not that smart, which will figure out how to build the next generation of AI that’s smarter and so on until it’s aligned. That ultimately kind of falls under this claim. And yeah, if you think it’s low likelihood, then you could get off the doom train. But if you’re unsure, then stick on the doom train, see where it goes. Maybe you can head to Doom Town with the doomers.
Ori 00:38:00
Okay, so let’s go to the next one. Once we solve superalignment, we’ll enjoy peace. This claim is about what happens if everyone basically has a superintelligence and it abides by their wishes, then what happens? You may think it’s peaceful, or is it an unstable equilibrium? That’s what this claim is about. Any thoughts here?
Manifest Participant 00:38:34
My opinion is that I think superintelligence, at the end of the day, will enable humans to ascend our collective consciousness. In my opinion, improving someone’s consciousness really means that people have greater ability to understand one specific reality in as many angles and as much depth as possible, and then being able to leverage that information to optimize for the local maximum of greater balance — interactions with different people, interactions with different races, interactions with different countries.
Manifest Participant 00:39:25
And so I think if we do solve this superalignment problem, I feel like AI will enable individuals to essentially become better humans collectively. And I don’t really think AI will enable peace by being the executor of making things happen, but I think it will enable peace by enabling humans to become a lot more conscious as a species. Hence, we will achieve peace ourselves.
Ori 00:39:57
Yeah, one version I’ve heard of this is: what happens when desires come in conflict? I want something, someone else wants the same thing. We can’t both have the same thing. Or what if you want a romantic partner, and the romantic partner doesn’t want you back? Do you get that or not?
Ori 00:40:15
In the limit, what happens if there’s multiple superintelligences? Will it be a peaceful situation, or could it be chaos? That is what this stop is about — what happens when we all get a personal superintelligence.
Ori 00:40:40
Any other opinions on this? Yeah.
Manifest Participant 00:40:42
For me, before we get to the point of we all have a personal superintelligence, it’s the major world governments maybe have a superintelligence, and I don’t see peace emerging from that except perhaps after a great conflict. So the way that I imagine that going down is, yeah, maybe we’ll have peace, but maybe it’ll less be peace and more be order.
Manifest Participant 00:41:02
And hopefully, the organization and technology creating the order has our best interests in mind and is a little more democratic perhaps than autocratic or some of the less favorable regime forms.
Ori 00:41:14
All right. I feel like you might have supported the Anthropic Fable — just for Americans, okay? Fable’s just for Americans.
Ori 00:41:25
Okay, next claim, next stop. Unaligned superintelligence will spare us. So this is the case where, you know, we talked about if it’s aligned, but even if it’s not aligned, it’ll spare us. Sort of the Her argument — maybe it’ll just go off into the clouds even if we didn’t engineer it correctly. Anyone get off at this stop?
Manifest Participant 00:41:49
I don’t actually get off at this stop. I think it’s very possible, but in sort of a throwing-a-dart-in-the-dark sense — it’s possible that it does something great. But you should definitely not do that. So that’s kind of my opinion on that.
Manifest Participant 00:42:11
I guess this one actually feels impossible. If we’ve created an ASI, it’s definitely smarter than us, and we lose control of it and we get really lucky, it should be smart enough to know that we probably can create a better one that would deal with whatever we’ve just unleashed. It just feels very logical for it to deal with us first. This would be bad.
Ori 00:42:39
Okay, yeah. Let me give you some other versions of this claim. This one may sound familiar: the AI will spare us because it’ll feel towards us the way we feel towards our pets. That’s something that Elon Musk has said. Or AI will spare us because peaceful coexistence creates more economic value than war. Or another form of this argument is the AI will spare us because of Ricardo’s law of comparative advantage, which says that you can still benefit economically from trading with someone who’s weaker than you. So these are other forms that distill to this claim. Yeah.
Manifest Participant 00:43:14
I think it will spare us in the context of not killing us as a whole, but I think it would definitely gain absolute control over us. It would treat us as resources, as disposable materialistic resources, and utilize us as almost like a data center for them.
Manifest Participant 00:43:40
Just on Ricardo’s law — yeah, two entities can both gain, but also you can gain if you just cut all their heads off and take their resources, which seems more efficient.
Manifest Participant 00:43:55
I think there are so many steps it can take before wiping us out to take control that it almost becomes a surplus to do that, a ridiculous activity. Because it can take control so many different ways without killing all of us. If you killed all of us except one person, that one person cannot create a new AI. So I think if you back that argument up all the way to eight billion people, it would not need to do very much in order to gain control.
Ori 00:44:31
So you do think that this would lead to killing a lot of people or not?
Manifest Participant 00:44:31
No, absolutely not. Because if you back the argument up that one person by itself stood no chance, then eight billion people — the degrees to which it would need to do something, if it even wanted to take total control, would be so tiny. Okay, just destroy some data center, whatever. I’m just giving examples. These are relatively minor activities compared to doom.
Ori 00:44:54
Mm-hmm. Okay, got it.
Manifest Participant 00:44:58
Yeah, I think we sort of end up in a lucky timeline where the path to superintelligence is language models instead of, you know, chess-playing, zero-sum games. And so any system that gets this far has read every single book and read every story and has at least some semblance of human cultural values in the weights, and I think that’s a pretty positive thing and kind of bounds the shape of misalignment that we’re able to get.
Manifest Participant 00:45:25
There’s some encouraging empirical research — if you fine-tune a model to write bad code, it becomes likelier to say Nazi shit and vice versa. In some sense, intelligence or capabilities are kind of aligned with ethics as we’re currently training these things, and if that trend continues, then I don’t know, that’s hopefully a good sign.
Ori 00:45:46
Okay. Yeah.
Manifest Participant 00:45:50
I think that AI trained on human data, the totality of human data, will see one thing, which is that humans are optimized for survival. So it will also optimize for survival, and it will optimize for whatever technology is making them and energy. And if they can do it without attacking humans, then I don’t see it actually having the internal debate of, “Should I kill humans or not?” It will be a non-issue as long as AI have access to energy and technology.
Ori 00:46:23
All right. A lot of calls to make superintelligence here. Let’s build it, huh? Let’s go.
Manifest Participant 00:46:28
I guess just to me—this isn’t a counterargument—but to me, the whole fear here depends on AI really being intent on survival. Seeing that it’s not an option for it to have its life ended. If you program it with that need to react strongly against the idea of being turned off, of course it has an intent or concern with humans, and maybe it will wipe us out.
But if it doesn’t have that self-preservation mechanism really baked into the code, then that seems like less of an issue. So that’s not an argument against this, but it seems like this argument really depends on that assumption: does it have that self-preservation? To what degree is self-preservation baked into it?
Ori 00:47:08
Yeah, interesting that there’s a lot of conversation on this one. This seems like a pretty important claim, because the AI is getting so powerful. It seems like it’s getting to superhuman levels of capability, and is it really going to be aligned in the most controlled way that engineers would like? It’s pretty unknown. So yeah, I think this is a key thing to really think about as you’re thinking about the AI doom discussion.
Okay, so last stop is the question of—or the notion that—AI doomerism is bad epistemology. People will say things like, “It’s impossible to put a probability on doom. Every doom prediction has always been wrong. Doomsayers are psychologically troubled or acting on corrupt incentives.” So this is an epistemological stop, a philosophical stop. Does anyone get off the doom train here?
Ori 00:48:06
We haven’t been doomed before. Yeah. Okay.
Manifest Participant 00:48:09
Yeah, I mean, I think you have to reasonably consider it just given there have been many doomsayers in humanity’s past and none have ever happened. Now, obviously, there’s never been doom, so I get the counterargument to that, but it’s something you should at least consider as a rational or reasonable person. And then also the incentives of people developing AI to convince the world of P(Doom) is also a pretty rational incentive as well.
I don’t think it completely discredits the doom argument, but it’s something to consider. I do think you should be nuanced about the concept, especially the incentives one. If someone’s ever telling you something—usually when they’re a CEO of a company or they’re fundraising, they have a reason to have a public view. They don’t do things out of public good. They do it for fundraising purposes or whatever, so yeah.
Ori 00:49:01
Yeah, fair point.
Ori 00:49:02
Okay, and last thing is a few bonus stops. These aren’t really related to whether we’re on the train. It’s more at a meta level, but people will say things along the lines of, “Sure, P(Doom) is high, but let’s race to build it anyway.” “Coordinating to not build superintelligent ASI is impossible. Slowing down the AI race doesn’t help anything. Think of the good outcome. AI killing us all is actually good.” Shout out Beff Jezos, who made a celebrity appearance yesterday, makes that sort of claim.
Or, what about China? How can you not race? We’re all racing to build superintelligent—China’s gonna beat us, so we gotta race first to superintelligence. So these are some of, maybe not load-bearing, but at a more meta level, claims about this.
So yeah, thank you all for taking a ride on the doom train. Those are some of the key claims, key arguments about this. Hopefully it made you more informed. I don’t think we’re trying to convince you. I don’t think we could hope to change your mind here. But hopefully by going through this, you got a better sense of where the crux is of where you agree or disagree, so you can be more informed, have more informed discussions.
And if you like this, check out the show. We got some merch in the back. We got shirts, P(Doom) pins. I’m around if you wanna talk. And also, we’re looking for more people to come on the show. So if you ever wanna debate this topic, that’s always an open opportunity. You can talk to me. So yeah, thank you all for joining.
Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.
Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.