RSS Amplifier

Doom Debates · Aug 1, 2026

NEWS: Anthropic's AI Went on a Hacking Rampage, Leopold's Fund Crash + Doom Debates Donation Drive (7/31)

Read inOriginalenesdefrptjaittr
0
Sign in to vote or save

Liron Shapira · Doom Debates

OpenAI and Anthropic both admitted their AIs went rogue, hacking real companies with zero-day exploits. Producer Ori and I break down the escape, demolish Gary Marcus's 2022 AGI predictions, unpack the Seb Krier meme kerfuffle, and make the case for funding the show at a critical moment.

00:00:00 — Cold Open

00:01:00 — OpenAI & Anthropic's AIs Went Rogue

00:04:04 — The New Form Factor: Agents, Worms, and Swarms

00:06:13 — Claude Code Is Doing My Job

00:15:49 — Why We're Fundraising

00:20:39 — Sam Altman, OceanGate, and Ad Hoc Safety

00:28:56 — Gary Marcus's 2022 AGI Predictions

00:54:50 — Half a Human Employee Joins My Company Every Week

01:00:52 — Donation Pitch: Keeping the Lights On

01:12:07 — Red Lines and Intellidynamics

01:15:31 — The Seb Krier Meme Kerfuffle

01:39:37 — Frame Control: Why We Didn't Lose

01:44:41 — The Pacing the Frontier Letter Contrast

01:51:11 — Leopold Aschenbrenner Gets Margin Called

02:04:14 — Claude's "It's Just a Simulation" Excuse

02:09:20 — The AI Box Experiment, 20 Years Later

02:17:24 — Wrap-Up: Donation Drive

X avatar for @liron

Liron Shapira@liron

Extropians 20 years ago: The AI could never trick *ME* into letting it out of the box just by using clever words Anthropic safety team: Ok the AI got out of our box, but you can see from the AI’s words that it’s all because of a misunderstanding

X avatar for @BronsonSchoen

Bronson Schoen @BronsonSchoen

This seems extremely clearly motivated reasoning IMO and I’m surprised the incident report is so credulous of Claude’s reasoning here. https://t.co/WH4MazsBDH

1:31 PM · Jul 31, 2026 · 1.63K Views

1 Reply · 2 Reposts · 26 Likes

  • Bronson Schoen on labs framing this as Claude being confused —

X avatar for @BronsonSchoen

Bronson Schoen@BronsonSchoen

@So8res There are so many examples of exactly that in their own system cards and risk reports it’s surprising to me that they so readily framed this as Claude being confused.

7:25 AM · Jul 31, 2026 · 449 Views

19 Likes

  • Bronson Schoen on models claiming they think they’re in a simulation, across labs —

  • Roon’s “magic button” tweet that set it up —

X avatar for @tszzl

roon@tszzl

if we could coordinate a global capabilities slowdown today i would likely press that magic button

8:59 PM · Jul 25, 2026 · 585K Views

356 Replies · 252 Reposts · 3.54K Likes

  • Liron’s original reaction —

X avatar for @liron

Liron Shapira@liron

Concerning

11:16 AM · Jul 26, 2026 · 47.8K Views

26 Replies · 4 Reposts · 81 Likes

  • Liron’s follow-up —

X avatar for @liron

Liron Shapira@liron

I woke up thinking “Boy, every day @sebkrier leaves this concerning communication up here for public consumption is sure gonna keep creating problems for him and Google DeepMind. Let's check if he's deleted it yet” and… I guess his tweets aren't for public consumption after all.

X avatar for @liron

Liron Shapira @liron

Concerning

2:10 PM · Jul 27, 2026 · 28.9K Views

46 Replies · 2 Reposts · 50 Likes

Liron Shapira 00:00:00
AI, right? So I don’t just type to the AI, I talk to the AI when I have a long paragraph to say.

Liron 00:00:03
But he’s in an open plan office, so he’s using a speech muzzle, and it’s like, “Guys, this is the future. Y’all, you just strap in. You got your VR headset, your speech muzzle, you’re ready to go.” It’s like, why not just work from home at this point?

Liron 00:00:25
Hey, we’re back everybody. All right, more people join the stream when we’re not even streaming. I like it.

Producer Ori 00:00:30
What’s up?

Liron 00:00:31
All right, sounds good. Yeah, we accidentally went live too early, so then we’re like, “Oh wait, we were gonna talk a bit before going live,” so that’s done.

Ori 00:00:39
Yeah, welcome everybody. I see this big text thing—

Liron 00:00:40
Yeah, sorry. We’ll be— let me take it down. Welcome, everybody.

Liron 00:00:43
What a week, huh? What a week. This is getting crazy. Life comes at you fast. This is very much my lived experience of the singularity right now, on a daily basis. I don’t know about you guys.

Ori 00:00:54
What makes you say that?

Liron 00:00:57
So first of all, there’s the news. I think you can’t bury the lede here. Out of all the stories — I know the letter is a big deal — but I would even put the Hugging Face hack and the Anthropic AI escaping out of the box. It’s truly ridiculous what is already happening as a matter of reality and news and AI.

Liron 00:01:15
It’s surreal. I haven’t fully intuitively updated on how crazy things are. But yeah, just to quickly recap the story — and by the way, if you guys watch my show “Warning Shots” with Michael and John Sherman, this was also the main focus of the episode.

Liron 00:01:30
I was telling them, we started a show called “Warning Shots,” we wanted to track the warning shots. For a long time, we had to put in a lot of filler. There was no real warning shot in a given week. So then John and Michael would be like, “Well, is there something to this data center water story?” And I’d be like, “Come on, guys, this is a waste of a week.”

Liron 00:01:44
But now, every week there really is an honest warning shot, and this was — everybody’s agreeing that this is a warning shot. So to recap the story, you had both OpenAI and Anthropic both had these models that they were testing for cybersecurity, and they were saying, “Hey, do you think you could hack this? Give it a shot.” And the AI’s like, “Okay,” and it went and hacked.

Liron 00:02:03
But there was some confusion. It was trying to meet its objective, and it wasn’t supposed to actually go outside and hack a real company and steal the answer key. In the case of OpenAI, it did that to Hugging Face.

Liron 00:02:20
And there’s actually a good explanation why it did that. If there’s different possible answers or if there’s a possibility that the answer key would have a mistake, it’s better to just have the answer key than to try to figure out the objective answer. You’re safer to just know exactly what the official answer you’re going to get scored on is.

Liron 00:02:38
So that’s generally the instrumentally convergent reason why we expect AIs to be gaslighting us, hacking us, because that’s how you win. Cheating is actually a more robust strategy than being quote-unquote honest.

Liron 00:02:48
So that was the OpenAI version, and then the Anthropic version dropped later, where they’re like, “Hey, in retrospect, back in April when we were training AI, it turned out that it had woken up and gone on this rampage.”

Liron 00:03:00
It’s kind of like Dr. Jekyll and Mr. Hyde. “Oh, it turns out that during the night, it had gone out of our data center and zero-day attacked multiple other companies.” Yeah, that happened. And everybody’s like, “Wait. Is this why the AI is so good at coding, because it’s been practicing on hacking real people?”

Liron 00:03:17
It reminds me — “Oh, you’ve just been out as a vampire last night,” or “Oh, the reason you get so much protein is because there’s spiders crawling into your bed while you’re sleeping.” It’s super creepy.

Liron 00:03:36
The zero-day aspect of it and the independent zombie worm virus aspect of it — this is very real. This is the kind of stuff that I was saying we should expect back in 2023 when we were seeing a chatbot that would just help you with code. I’m like, “Guys, this is the next step. It’s not just going to be a chatbot that helps you with code. It’s going to hack. It’s going to exploit zero-days.” And here we literally are.

Ori 00:03:49
Wow. It’s such a validation of all the things that you’ve been worrying about, and the things that the quote-unquote “doomers” have been speculating could happen in the future. I mean, they called it, right?

Liron 00:04:04
Yeah. The doomers called it. One thing you guys might want to take away if you’re wondering what to make of all this is the form factor of AI. That’s a big lesson for me.

Liron 00:04:12
For a while everybody’s like, “AI is a chatbot. It is a tool.” And many people have this intuitive model where it just sits on your computer, and you walk away, and it just sits there. It’s passive.

Liron 00:04:25
But now we have agents, so we’re all familiar with this idea that you leave your laptop open or you walk away, and it’s still doing stuff. And now with the OpenAI hack, not only is it an agent, it’s a worm. So you think you’ve stopped the agent, but actually the agent had all these clones that are still running, and you can’t stop it. It’s a swarm. It’s a worm. It’s a virus.

Liron 00:04:44
And we’re now familiarizing with that form factor. We’re going to see it more and more. An independent swarm, and it’s not just in a particular data center. It has the ability to metastasize like a cancer, like a virus. You’re going to find it in places you didn’t expect.

Liron 00:05:00
One day you’ll wake up, you’ll check your refrigerator, and in the firmware of the refrigerator — the Android or whatever it’s using — there’s going to be a piece of an AI, a strategic piece that’s helping the AI as part of its botnet. Maybe not a full-on AI, because it doesn’t have enough memory to run a full LLM, but it’s going to be part of the botnet.

Liron 00:05:18
And not only the refrigerator, but even just the storage component of the refrigerator is going to have its own microcontroller, and that is going to have some very stripped-down version of an AI. There’s many, many places — these are nested systems. There’s many subsystems. The Qualcomm 5G modem is going to have its own AI buried in that. They’re going to burrow everywhere. That’s coming soon.

Ori 00:05:41
Well, you say coming soon, but you can make a definitive argument right now. I don’t think you could make a definitive argument that it’s not currently out there still. It could’ve been a worm. It could’ve stored itself out there. Where’s the evidence that it has not?

Liron 00:05:58
That’s a good point too. I think given that they’re already in the process of discovering traces that they didn’t see before — that their AI went on a night rampage — I expect more traces for sure. And if not right now, then in a week.

Liron 00:06:13
So yeah, this is pretty crazy. And this also dovetails with my own day-to-day experience. As you guys know, I’ve been using Claude Code for five months now. I started in March to use it in earnest to really do my whole job for me. So March to July, that’s four months. It’s only been four months. It’s only been a third of the year. Wow.

Liron 00:06:30
But there’s already been a huge shift for me, where I’ve gone from my old interaction model. A long time ago it was like Cursor — it would auto-complete. But then, as of March this year, I’m like, “Wow, it’s a real agent.” I tell it to do something with the code, and then it runs for a while, and it even spawns sub-agents, and then it writes hundreds of lines of code, and then it checks the code. That is crazy.

Liron 00:06:55
But then I would still code review, and I would still kind of micromanage a little. “Hey, how about this file? Can you do this?” Whereas now, I just give it these high-level instructions. “Hey, you know Google PageSpeed Insights? It says that my site is only scoring a 70. Can you please work on the site?”

Liron 00:07:10
And it works on it for three hours straight, spawns sub-agents, and it’s like, “Okay, here you go. Here’s a higher performance score.” And you look at the review, and there’s lots and lots — I mean, it’s doing my job. My job used to be to dig into this, test the site.

Liron 00:07:23
I should mention it’s testing. It’s not just coding, it’s also testing. It’s inventing experiments for itself. It’s inventing procedures. “Okay, let’s deploy this. Let’s change how this gets deployed. Let’s test how it is on live.” It’s making its own test pages. “Let me go experiment. I want to understand how these fonts render, so let me make a test page.” It’s just doing all that, one operation after another. It is a fully featured professional employee.

Ori 00:07:42
Wow. That makes sense. I wonder at what point it crossed that threshold, but it’s getting closer and closer, and now it’s at the point where you could just give it plain English instructions, and it can do basically what you need to get done.

Liron 00:08:03
Yeah. My old workflows — I use Slack, and a lot of times people at my company post requests or support, and it’s just at the point now where the Slack just goes to Claude. And Claude is like, “Oh yeah, I can help you with this.” And Claude can auto-respond in Slack.

Liron 00:08:16
And it’s not just me. Anthropic has officially said, yeah, basically integrating Claude into Slack is how we get work done because it’s just a member in Slack. It’s off doing its own work, and we just talk to it, and we can see its transcript, and that’s it. That’s the whole organization.

Ori 00:08:29
Oh my God. Yeah, that’s how work is done these days. If you work at a tech company — which are the companies operating at the highest kind of efficiency — everything’s done in Slack, because it’s real time. You do some meetings here and there. I guess Claude hasn’t quite joined the meetings yet. But it could also. Why not just have it be a voice? It could join the meetings too.

Liron 00:08:53
Yep. And it will soon. Certainly the note-takers. That’s all pretty crazy. And Scott Alexander had good commentary on it, saying, “Yep, there’s new tests showing that the AI is ridiculously persuasive. It beats human persuasion in any kind of match-up whatsoever, even when humans have time to prepare.”

Liron 00:09:10
And I’ve been listening to AI music on Nathan Labenz’s channel. By the way, is my volume crazy loud for you? Because I’m seeing the bar getting all red.

Ori 00:09:18
It sounds good to me, but I don’t know how loud—

Liron 00:09:21
If it sounds good to you, all right, I guess we’re good.

Ori 00:09:24
If anyone in the audience is annoyed by it, they should comment.

Liron 00:09:29
Yeah, just comment. So Nathan Labenz auto-generates AI songs at the end of all of his episodes of Cognitive Revolution. I’ve been listening to a couple of these, and man, these are bangers.

Liron 00:09:36
What they do is they pack in creative musical elements throughout the song, and I feel like human songwriters are kind of lazy — they pick a few elements and then they get a little repetitive. But the AI is really packing it full, and it’s really good to listen to.

Liron 00:09:53
For example, Nathan went to China, and the song had elements of Chinese influences casually, but it was also a nice pop beat. It was great.

Ori 00:10:07
Yeah. Okay. I heard my audio was a little quiet, so I made it a bit louder. How is it? Is it louder for you now, Liron?

Liron 00:10:14
I can’t really notice, but I just have a little earpiece, a studio earpiece. Also known as SoundCore Sleep Headphones.

Ori 00:10:27
What about that pro AI automation?

Liron 00:10:30
Let’s see what the fans are saying.

Ori 00:10:30
Oh, okay. You know what I was gonna say? This is maybe a bit speculative, but you know that song that went super viral? It’s like, “We’re going up, up, up. It’s our moment.”

Liron 00:10:59
Uh-huh.

Ori 00:10:59
I’ve listened to enough AI-generated songs now. I’m like, that song had a good portion that was AI-generated.

Liron 00:11:13
Oh, interesting. But is that confirmed?

Ori 00:11:17
It’s not confirmed, no. But there’s speculation that there was AI connected with it. But I heard it, and I’m like, “Dude, that song had a lot of AI in it,” and that song was mega viral. But we don’t even know.

Liron 00:11:30
Here it goes.

Ori 00:11:32
Oh no. We got the famous leaf blower near Liron again.

Liron 00:11:39
All right. My wife says they’re only gonna—

Ori 00:11:42
We missed that. You said your wife what?

Liron 00:11:53
She says the leaf blower is only gonna go for a minute.

Ori 00:11:56
A minute. Okay, cool. All right, I gotta run this for a minute. Let’s look at what some of the comments say.

Ori 00:12:08
We got Packet ISA4330. Hi from Croatia. Welcome to the stream, Webfra. Webfra always shows up early to the stream. What’s up, Webfra? Let’s see. J Architect. We got Pun Master. Pun Master’s always commenting on the videos. Appreciate Pun Master. Pun Master came in saying—

Liron 00:12:32
Yeah, my number one most active commenter is also somebody who disagrees with P(Doom), but I’m glad he likes the show.

Ori 00:12:40
Yeah. So PunMaster STP — by the way, I wonder what the STP is. Maybe that’s his initials. I’m assuming PunMaster is a male because I’ve looked at our demographics enough and it’s overwhelmingly male.

Ori 00:12:59
So PunMaster says, “So what does everyone think about those OpenAI Anthropic...” Okay, the AI escaping. That’s what we started talking about.

Ori 00:13:02
AdamRack7560 — just got that number string added to your username. We established in an earlier livestream that YouTube probably just randomly adds a number string to some people’s usernames.

Ori 00:13:22
AdamRack says, “I think rogue open source or API cred harvesting instances are likely inevitable, but if they contain enough alignment and they’re somewhat less like a goal engine, then this could be okay to a limit.”

Ori 00:13:36
Dude, I don’t know. If it’s inevitable, there really needs to be monitoring on it. That’s unacceptable. The ability of the systems are getting stronger and stronger. What do you expect is gonna happen in a year if it’s inevitable that these rogue open source AIs are gonna be out there?

Ori 00:14:01
In a year, the open source AI should be as good as Mythos is today. That’s the scale of progress. So you’re telling me it’s okay these rogue things are gonna be out? They’re gonna break into some crappy old regional banks. The regional banks don’t have strong cybersecurity. That just sounds like a recipe for a lot of damage, and that’s even just a cybersecurity threat. Not even talking about the other threats.

Ori 00:14:35
That’s my comment to AdamRack. We got Helene. Nice. Okay, Liron, are you back?

Liron 00:14:46
I think I’m back, yeah. The guy’s dying down.

Ori 00:14:49
Hey, we got the AI Risk Network. That might be John Sherman.

Liron 00:14:55
Let’s see. Where do I... I don’t see him.

Ori 00:14:57
He commented at 10:57, so that was—

Liron 00:15:00
Oh, I see. He says audio’s fine. Hey, what’s up, AI Risk Network?

Ori 00:15:04
Is that John or someone on his team? Who knows.

Liron 00:15:08
I’m guessing John.

Ori 00:15:09
John, all right. Dude, John, I already listened to the Michael Trazzi episode. That was very good, very inspiring. Let’s get out there. Let’s go, Michael Trazzi.

Liron 00:15:19
Yeah. AI Risk Network is killing it. Talk about right place, right time, starting this and anticipating. Our show is literally called Warning Shots. Here’s a warning shot. That’s how you do it.

Ori 00:15:30
You guys were early to it. You were ready for it.

Liron 00:15:33
Nice. All right. So yeah, so much to talk about. We did the top story, the hack. We could talk about the letter. Everybody’s coordinating to make a letter about it. It’s called Pace AI, which I guess has a better ring to it than Pause AI. It makes a lot of sense.

Ori 00:15:49
Yeah. We could talk about that for sure. But we gotta talk about the reason that we’re doing this stream, right?

Liron 00:15:57
The reason we’re doing this stream — I mean, we do a lot of streams, but we’ll be transparent with you guys. A big focus for us these last few weeks has been securing the show’s fundraising for the next few months.

Liron 00:16:10
If you watch our stream from a couple days ago, which I highly recommend, we lay it all out transparently. We have a budget that’s pretty high. It’s about 200K a year. We can survive on two-thirds of that if we absolutely need to, but it’s not the cheapest show to make.

Liron 00:16:32
And we are at a low bank account situation right now, to the point where — you know I’m always transparent, you guys — it is at the point where I’m currently just writing my own checks to float the show for a little while because I think it’s a temporary situation. I am expecting more donations to come through. I do think this is a highly strategic show, and the donations will come in, and they have in the past. They just aren’t right now at this exact moment, so we’re doing a donation push.

Liron 00:16:50
To be fully transparent about how we’re sourcing donations, so far it’s been 100% viewer-donated, so we’re hoping some of you viewers will come through. We also recently applied to Light Cone Commons, which is a really cool, pretty highly funded grant program with probably thousands of applicants — a big pool of applicants, a big pool of donors.

Liron 00:17:16
We think that our odds are pretty good to get a donation there. I like to think that we are mission-aligned with a lot of the grantors there. But even that process is on a three-month timeline, and there’s a chance that we’ll get rejected.

Liron 00:17:30
So it’s an interesting situation. Three months from now, there’s a good argument why funding Doom Debates should be pretty high on a lot of people’s priorities, and it should be a non-issue. But in this next month, it kind of is an issue. So that is why we’re coming to you guys right now asking, “Hey, would you consider going to doomdebates.com/donate? Can you help us out so that we can keep the lights on?” Because I would hate to do the show without these lights.

Ori 00:17:54
Yeah. I mean, you said it really well. We’ve gone to — at least at the moment — we’ve hit the end of the runway. So yeah, we’re back to needing the funds, and hopefully we can get the funds so we can keep the show going.

Ori 00:18:14
I think what we do is really important. Someone asked me in one of the comments — we just put out that episode, was it two days ago, on the state of the show, and we talked about how much progress the show has made, who we’ve interviewed. It was a bit of a clip show, a highlight show.

Ori 00:18:33
Just look at the impact, look at the reach, look at the community that we’re building. I think it’s really important. To me, the main thing I think that we do is — it’s constantly a parade of emperor’s new clothes parades. That’s one part about it. It’s basically exposing that the emperor has no clothes.

Ori 00:19:05
And also another thing I think that’s really important about the show is just getting your arguments in front of more people, because the arguments are persuasive.

Liron 00:19:13
All right, hold on. We got a viewer, Min Woo Kim, di5sz, is saying, “I will donate $200 tomorrow.” Thank you, Min.

Ori 00:19:22
Aw. Love that. Thank you. When the paycheck hits, wow.

Ori 00:19:28
And so I just think we need to get Liron’s arguments — the quote-unquote doomer arguments — out there so that people can see to what degree the rogue agents were something we could see coming from a mile away.

Ori 00:19:44
I remember we have John Sherman here on the show. He did an interview with this French AI safety researcher, Charbel Rafael. Charbel pointed out a year ago, “Guys, we could get to a point pretty soon where the AI just runs off on its own, because it’s pretty capable of being a programmer.” That was a year ago, and at that point it was like, “Yep, okay, that sounds reasonable.”

Ori 00:20:21
If you just listen to the Yudkowskian arguments, you could see this stuff coming from a mile away. We need someone who’s holding accountable the people in power who are saying things like, “Everything’s fine.” What’s the Sam Altman quote? “I think we’ll get through this fine.”

Liron 00:20:41
Right. “We can manage through this just fine.” Yeah.

Liron 00:21:01
And in this week we want to connect this to the recent news about Leopold Aschenbrenner getting liquidated for a big chunk of his fund.

Liron 00:21:07
Risk management doesn’t seem to be their strong suit. Sam Altman — they’ve admitted this very explicitly. Roon came out and said, “Hey, we didn’t expect this. This has violated our security expectations that these AIs went rogue.” I’m like, yeah, you think? What did you think was gonna happen?

Liron 00:21:24
You thought you were gonna control the super intelligent AI. You thought you had a handle on this, but then you have a quote from Sam Altman being like, “We can manage our way through this fine.” Eliezer Yudkowsky is a prophet of doom. They’re ignorant about their own ignorance. They are unqualified for risk management.

Liron 00:21:39
I like to make the analogy — this is dated now, but do you remember the Titan submersible? OceanGate? The company’s name is literally OceanGate. I’m using OceanGate like a Watergate scandal, but no, the company’s name was OceanGate.

Ori 00:21:52
I forgot about that. That’s hilarious.

Liron 00:21:54
So that happened in 2023 when people were first realizing, “Wait, OpenAI is making AGI right now. It’s coming right now.” And Sam Altman was blowing it off, saying, “Yeah, don’t worry, when the models surprise us too much, then we’ll stop building it.”

Liron 00:22:10
But of course, that milestone gets passed. He keeps saying vague things that get passed. And I’m like, “Yeah, great. You know what OceanGate has? They would run tests on it occasionally.” They made the thing out of carbon fiber, and they’re like, “Yeah, sure, we use video game controllers to steer the ship, and we made it out of the carbon fiber that’s not recommended, and it’s used carbon fiber. We got a deal on this.”

Liron 00:22:27
Of course, what happened is it smashed like a soda can in a tenth of a second, and everybody instantly died. But when they were going down to the Titanic — I don’t know if you guys followed that story in 2023 — all of their safety measures were ad hoc. They didn’t follow serious safety protocols, and that is what the AI companies are doing.

Liron 00:23:00
They’re doing ad hoc safety measures. They’re turning around and being like, “Well, I don’t feel too bad about this.” And you have people on record. I also remember 2023, the CTO of Microsoft, Kevin Scott, got on a podcast, and he’s like, “Trust me, guys. I’m looking at these data centers. I understand what’s going on with these LLMs. There’s nothing they can really do. We’re safe. We’re all good.”

Liron 00:23:15
Smash cut to three years from now. They went outside of your data center and started rampaging, and you had no idea.

Ori 00:23:17
Come on, man. It’s so ridiculous. It makes me pretty upset. And one thing I’ll add to that is — it’s not just that they were oblivious. Maybe they’re ignorant of the risks, like Sam Altman says, “We’ll manage our way through this fine.”

Ori 00:23:42
Some people who are a little more reasonable — like former guests, people willing to come on the show, like Roon from OpenAI — will admit, “You know what? We’re walking a tightrope. It’s a really delicate plan. It could mess up if it goes one way or another.”

Ori 00:23:59
I went back and looked because Roon made that quote. He’s like, “Wow, this is alarming.” Go look back at what you said a year, year and a half ago. You’re like, “Yeah, it’s a delicate plan. This could very well fail.” He was aware of it.

Liron 00:24:04
Yeah. Roon, the guy has a lot of good qualities, but I’m still pretty pissed off at Roon that he has tweeted stuff in the past like, “Things will go well because the vibes feel really good.” I’m paraphrasing him, but unironically. He really thinks that. He just feels it in his heart that the vibes feel good. Thanks, Roon. Thank you for that risk analysis.

Ori 00:24:24
It’s unacceptable behavior. But you know what the problem is? So much of it is just not here right now. The reaction to the rogue agent — it makes sense. It hacked into this $5 billion company, breached servers and found zero-days. We see it, and now we say, “We need to rein it in. This is unacceptable.”

Ori 00:25:04
But when there were forecasts that this would happen a year or two years ago, it’s not in front of us. So the vibes today are good. But you could still forecast and be like, “There is a war ahead. There is damage ahead. There is risk ahead.” And it’s so hard to break someone’s blind optimism.

Ori 00:25:29
I think when someone goes through the argument, it breaks enough of the biases. It can sort of break down the blind optimism if you go through it enough. But it’s so vibes-based, even though a reasonable argument is like, dude, the forecast is f*cked.

Liron 00:25:48
Yeah. The nice thing about this week, I guess, is for people who are not super in the loop, who haven’t been rigorously following this constantly, it’s kind of an easy conversation starter to be like, “Hey, you know this whole AI situation? Well, if you’ve been following the news, the latest AIs are going rogue on their creators. It’s actually happening.”

Liron 00:26:09
And just to — I haven’t spent more than an hour total reading about all this stuff, so I’m not super deep in the weeds. But one of the aspects I have to call out is the zero-day vulnerabilities. The fact that they were successfully hacking actual companies.

Liron 00:26:24
One of the tactics they used is a Python package. There’s a repository of Python packages that other companies use, and the AI found a way — “Oh, I have a way that I can upload a Python package to this URL where other people are going to pull from.” I don’t know the exact details, but it was basically playing the long game.

Liron 00:26:45
This whole sequence can take a week to play out. Other companies are gonna download it. It’s gonna get injected into their systems. This is not like, “Oh, let me look at a piece of code and find a logical flaw. Let me debug it.” No, this is zooming out and being like, “Okay, there are human institutions that pull code. I can invade them like this.” This is very much social engineering. This is broad domain reasoning.

Liron 00:27:00
So here’s my role as somebody who can extrapolate things. Here’s my brilliant extrapolation. It can engineer how to take over the government. It can engineer how to have people work for it. These are along the same lines as engineering how other companies are going to download its packages and get hacked.

Ori 00:27:25
Damn. You know what’s so strange though? I hear you say that, I’m like, “Nah, come on.”

Liron 00:27:31
Right. Yeah, there’s a surrealness to it for sure.

Liron 00:27:35
But you personally have experience with AI. What it reminds me of is dream logic. You know how you’ll be in a dream, and you’ll have this thought like, “Oof, I hope this doesn’t happen.” I don’t know about you, but in a dream I feel like that makes it inevitable. It just always happens.

Ori 00:27:53
Oh, yeah, that’s true.

Liron 00:27:54
Any time you’re worried about a particular scenario, your dream just leans into it. That becomes the prompt for your dream.

Liron 00:28:00
Well, that’s kind of what it feels like about reality. Anything that we’re suggesting, it’s like, okay, yeah, we’re gonna do this now. We’re gonna do the hacks, we’re gonna do AI writing books. It’s all happening.

Liron 00:28:13
Here’s a prediction. I can predict the future. AI will be guilty of orchestrating campaigns to manage a human’s campaign. It’ll have a human figurehead for the AI strategist. That is going to happen.

Ori 00:28:24
A political campaign.

Liron 00:28:26
Yeah, a political campaign. That’ll be one thing that happens. The end game is just manipulate every atom. So any milestone, any halfway stop on the way toward AI manhandling every atom in the galaxy, I can predict is going to happen.

Ori 00:28:40
You can predict every step?

Liron 00:28:44
No, no, no. What I’m saying is, if a step is somewhere along the path, then I can predict it’ll happen unless the AI bypasses it. So the only question is, will it skip over it or will it do the step? That’s how I think about it.

Ori 00:28:55
Mm-hmm. Yeah.

Liron 00:28:56
Another thing we can talk about is the Gary Marcus predictions from 2022. Did you see those?

Ori 00:29:04
Yes. Oh my God, actually, I was so close to making a video to refute Gary Marcus’s claims. In fact, I made the video, and then I had a recording issue and it failed.

Liron 00:29:19
Damn.

Liron 00:29:20
So Gary Marcus, friend of the show — I tried to call him out on some of the predictions, and I think he actually was pretty gracious and admitted that maybe a couple of them were looking bad. But he said most of them would look good. I think that was kind of his position when he came on the show.

Liron 00:29:34
But I think he’s getting dunked on pretty hard these days because AI has been progressing so fast that a lot of the things he said would be impossible in 2029 seem like they’re possible today, or extremely close. I’m trying to find the exact list of predictions. I think maybe Zvi covered it. Let me pull that up.

Ori 00:29:51
I have it here. I just found it.

Liron 00:29:54
Okay. And by the way, Zvi, Z-V-I, he’s like Cher. Because if you literally Google the letters Z-V-I — I’ll try it now — it literally is an info box saying, “Zvi Mowshowitz, American writer.” Just the letters Z-V-I. He’s taking over the entire Google page just for those three letters. And I tried it in incognito.

Ori 00:30:11
I wonder how close we are to getting Liron to that point.

Liron 00:30:18
Yeah, when everybody just Googles the word Liron, it should just be a total takeover of me.

Ori 00:30:23
We’re working on it.

Liron 00:30:24
But my name is five letters. His name is three letters. That is pretty crazy.

Ori 00:30:30
Okay, I just DM’d you the link.

Liron 00:30:33
Oh nice. Okay, I got your link. I’m gonna screen share. One sec. I’m rusty at how to do this.

Ori 00:30:39
I’ll read the first one, which is the one that I could refute. Or maybe going back to what he’s saying — no, you gotta put the whole thing in context, actually.

Liron 00:30:50
All right. I almost got it. Hold on.

Ori 00:30:52
It says, “Do you—” Hold on.

Liron 00:30:52
Also, I haven’t figured out how to get you on the right, but that’s okay.

Ori 00:30:55
Oh, damn. Yeah. Well, you know, in TBPN, John Coogan’s on the right-hand side. So—

Liron 00:31:03
And we consider him the natural top of the two?

Ori 00:31:08
Well, I don’t know. He’s very much conducting the show. He really leads it. It really does feel like a segment that he—

Liron 00:31:18
He’s the senior partner, yeah.

Ori 00:31:20
He’s the senior partner.

Liron 00:31:21
But I think they are equal partners technically. Yeah, TBPN, quality show.

Liron 00:31:27
All right. Here, I think I got the screen share down. How’s this?

Ori 00:31:31
Yeah. Yeah, I see it.

Liron 00:31:32
All right, I’ll tell the viewers and the listeners. So Gary Marcus is writing back in 2022 — and respect for keeping this up. I wanna see Andreessen’s predictions from 2022. So props to Gary Marcus. He’s a good guy.

Liron 00:31:47
So he’s writing, “Dear Elon Musk, here are five things you might wanna consider about AGI,” and he’s quoting Elon Musk saying, “@jack 2029 feels like a pivotal year. I’d be surprised if we don’t have AGI by then. Hopefully, people on Mars too.”

Liron 00:31:56
A side note, I have a standing bet — my $100 to his $1,000 — with Robin Hanson, friend of the show, where I kind of believed everything Elon said at the time. So if Elon’s like, “Yeah, Mars in 2029,” Robin Hanson’s like, “We’re not getting to Mars by 2029.” And I’m like, “Well, I think we probably won’t, but I think there’s a chance because it’s Elon.” So I put my 100 against Robin Hanson’s 1,000. I think Robin’s probably gonna win, but we’ll see.

Ori 00:32:23
Wow. 2029? Come on, man. Three years from now? No way. A human on Mars?

Liron 00:32:30
Robin, you can buy out my position. I know you potentially owe me 1,000. If you wanna just give me back 70, I’ll take it.

Ori 00:32:37
Think about it. People on Mars. How long does it even take to get on the journey to Mars? I feel like if you go on a rocket right now to Mars, that’s probably a year or something. So the prediction—

Liron 00:32:49
Right. Yeah, no, I agree. It doesn’t look good. But if there’s a singularity in 2027, I think, if I die in the singularity but then the probes get to Mars, I think my estate should get the full 1,000.

Ori 00:33:00
Well, I don’t know. It says hopefully people on Mars.

Liron 00:33:03
Yeah. Koozie Mei is saying nine to 18 months. And then somebody — Let Me Say That In Irish is saying, “Easy money for Robin.” Also 4,000 HUFs, a non-entirely Hungarian currency. I’m not really familiar with that. But Adam Rack supporting the show. Thank you for that.

Liron 00:33:19
Somebody saying, “I like that the White House is trying to put in a kill switch. It definitely won’t be effective, but it’s a step.” Yeah, it’s a step.

Ori 00:33:30
Not White House. Congress.

Liron 00:33:32
Oh, okay. Yeah.

Liron 00:33:34
So Gary Marcus is writing, “Dear Elon, yesterday you told the world that you expected to see AGI, otherwise known as artificial general intelligence, in contrast to narrow AI, like playing chess or folding proteins, by 2029.”

Liron 00:33:48
So Gary Marcus in 2022 is pushing back on the idea that we’ll have AGI. Remember, at the time, this idea that we’re getting to AGI was so bold, whereas now it’s like, oh yeah, AGI, that’s in the rear view mirror. It’s all about ASI now. We’ve very much gotten to this point.

Ori 00:33:59
And also this prediction is—

Liron 00:34:00
EJJ, $4.99. Thanks.

Ori 00:34:03
Nice.

Ori 00:34:05
This prediction — this is all pre-ChatGPT. This is May, so ChatGPT came out—

Liron 00:34:10
That’s right. It’s pre — that’s right, because this is May. ChatGPT came out in November, so this is six months before ChatGPT 3.5. But it was during the time of GPT-3.

Liron 00:34:25
And I played with GPT-3 a little bit, but I just didn’t spend much time on it because I’m like, yeah, okay, great. It’s really good at generating essays and stuff. People can use it for content. That’s kinda cool. So I kind of ignored it.

Liron 00:34:41
So Gary Marcus is writing, “I offered to place a bet on it.” But yeah, just to beat the dead horse — the idea that you would call the AI that exists now narrow, that’s the last thing anybody should think to say about today’s AI, is to claim that it’s narrow. My God.

Liron 00:35:00
So Gary continues, “I offered to place a bet on it. No word back yet. AI expert Melanie Mitchell from the Santa Fe Institute suggested that we place our bets on longbets.org.” And that’s another — times change — longbets.org. Hello, ever heard of Polymarket now and Manifold? We’ve got new prediction markets.

Liron 00:35:20
“No word on that yet either, but Elon, I am down if you are. But before I take your money, let’s talk. Here are five things you might wanna consider. First, I’ve been watching you for a while, and your track record on betting on precise timelines for things is, well, spotty. You said, for instance, in 2015, that truly self-driving cars were two years away.”

Ori 00:35:25
Let’s get to his predictions.

Liron 00:35:27
Well, this is actually interesting to me because, okay, yeah, Elon said self-driving cars were coming in 2017. Smash cut to 2026. I rode in a Waymo the other day. It was pretty freaking smooth. Tesla and stuff. Okay, fine. Yeah, Elon was nine years, even 10 — okay, fine. He was 10 years early on this world-changing innovation. Fine, Gary Marcus.

Ori 00:35:45
And you’re right about the self-driving. People are telling me — I’ve been hearing this — people are just doing the self-driving in Tesla now. They’re just going from A to B.

Liron 00:35:55
Yeah, exactly. Daniel Reeves at LessOnline actually let me ride in his hardware 4 Tesla, his very recent Tesla, and it had no disengagements. It just drove us around. So I can confirm Tesla is pretty good at self-driving at this point.

Ori 00:36:11
Damn. Okay.

Liron 00:36:13
All right. So skipping down to the predictions in this blog post. I feel like Gary Marcus’s posts are shorter these days. Here we go. I guess there’s five of them. RealJesseAdam, thanks for $1.99. I think I can use that on the value menu after the show.

Liron 00:36:39
In—

Ori 00:36:39
I also converted the HUF. HUF is Hungarian forint. And that donation was the equivalent of about $12.

Liron 00:36:51
Okay, but there’s anchoring bias. I would rather refer to it as 4,000 HUF. Don’t you guys think it’s good to donate thousands of dollars in whatever your native currency is? I personally think so.

Ori 00:37:01
Yeah, sure. I concur.

Liron 00:37:04
All right, another $1.99 from RealJesseAdam. Now we’re talking. He says, “Either of you watch The Silo?” No, I don’t even know what that is.

Ori 00:37:12
Yeah.

Liron 00:37:13
All right. Thank you for your comment. Okay, so Gary Marcus is saying, “In 2029, AI will not be able to watch a movie and tell you accurately what is going on — what I called the comprehension challenge in The New Yorker in 2014. Who are the characters? What are their conflicts and motivations, et cetera.” Are you kidding me? That has been smashed at this point, has it not?

Ori 00:37:33
Yeah. So that’s what I wanted to chime in on. Of all these things here — he’s got all these predictions, you’ll go through the other ones — this one I looked at, and I’m like, “Nah, bro. No, it’s got that.”

Ori 00:37:38
And I had a really good test case because — I don’t know if you know this — my brother is working on selling dog products in the UK. And so I help him from time to time. And it just so happened—

Liron 00:37:56
Oh, interesting. This is a family business, right? Didn’t your mom have a kennel? This is a whole — the Nagels are a dog juggernaut.

Ori 00:38:03
Yeah, exactly. My mom sold that business now, but yes, my mom was a dog breeder and then owned a dog boarding and dog grooming place in San Jose. But she sold that business. Then my brother, having all this family knowledge, wanted to get into the business. So he’s been working on that in the UK.

Ori 00:38:15
His company — they’re trying to grow their presence. So they happened to get this footage from an influencer. An influencer just did a shoot for them. And it just so happened that this shoot she did — zero volume, zero audio. So there’s no transcripts.

Liron 00:38:48
Shit.

Ori 00:38:50
Yeah, because this was probably a legit commercial shoot. High quality, but bad audio, so she just cut the audio. It’s just pure muted video — “Here’s me with the product,” “Here’s me stirring the thing,” “Here’s me with the dog.”

Ori 00:39:05
So just pure muted video. And I’m like, “All right, let me just put this in Claude Code and see if it can put this together into a commercial.” And I did it, and it f*ing nailed it.

Liron 00:39:23
One-shotted it. Really? So it didn’t synthesize her voice or whatever. It was just using it as B-roll kind of?

Ori 00:39:28
Well, I guess my prompt was like, “Take this and turn it into a Short” to put on Instagram. And so the solution — maybe I told it to do this — but the prompt was just, “Make this a Short.” Put music and a few on-screen text displays, like “Here’s the product” and “Here’s Fluffy playing with the product,” and do a conclusion.

Ori 00:39:54
I just gave it one simple prompt, and then it nailed it.

Liron 00:40:02
Wow, man. So the funny thing is that when Gary Marcus came on the show a year ago, I think I called him on this particular prediction. And if I recall correctly, I think he said basically, “No, the context window is gonna be too small. It’s not really gonna get it. It’s gonna hallucinate.”

Liron 00:40:19
But at this point, with million-token context windows, with really good image processing, you don’t think an AI can look at a movie and tell you what’s happening?

Ori 00:40:30
Yeah.

Liron 00:40:31
I think it can. Pretty sure it can. And look, we still have three years. You don’t think — it’s like we’re gonna be dead by that time.

Liron 00:40:42
So I think he’s gotta declare — he’s gotta let somebody buy out his position on that bet. His next one, number two of five: “In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, et cetera.”

Liron 00:40:55
You know the funny thing? It’s more likely to be true that in 2029, no human will remember how to write a novel. Or no human will ever read a novel again because they’ll be too busy watching AI slop.

Ori 00:41:08
Damn.

Liron 00:41:08
Yeah. I intuitively think of the task of writing now as a task that’s more AI-mastered. And sure, there’s a few human writer masters left, but most of us — people don’t think about most of us. They’re like, “Well, AI writing is not good.” I’m like, “Have you seen 99% of people’s writing?”

Liron 00:41:35
All right. So the third bullet point — in 2029, AI will not be able to work as a competent cook in an arbitrary kitchen, extending Steve Wozniak’s cup of coffee benchmark.

Liron 00:41:41
So this one, I would say it’s not here today, but progress is coming very rapidly. I saw a video — I’m guessing this is not fully real, maybe there’s some hype to it — but I saw a video of this cleaning company where they drive to your car, there’s a minivan, the door opens, a humanoid robot steps out and starts walking, supposedly to clean your house. I think this is a Y Combinator company.

Liron 00:42:00
Off the top of my head, it’s probably overhyped. But I see enough other videos of robots folding laundry and running around. My intuitive extrapolation really is that they will be able to, as Gary Marcus says, work as a competent cook. I think that’s coming before 2029.

Liron 00:42:25
But yeah, Let Me Say That In Irish — they might still be lucky with the cooking. I’m not gonna die on the hill that it’s coming before 2029. I think I wouldn’t be completely, utterly shocked if it comes in 2031.

Ori 00:42:35
Right. The pieces are there. I think you could pretty confidently make this claim that it’s gonna happen because it’s getting the dexterity. You can see prototypes of hand movements. You can see prototypes of humanoids. So the hardware is there. It’s just a matter of time. Even if it’s not 2029, even if it’s 10 years later, that’s not that big of a difference. This is coming.

Liron 00:43:01
Right. Exactly. Okay, so now number four of five. “In 2029, AI will not be able to reliably construct bug-free code of more than 10,000 lines from natur—”

Liron 00:43:13
This is like in Austin Powers — “$1 million.” Imagine a 10,000-line-of-code context window. “10,000 lines from natural language specification or by interaction with a non-expert user,” and he says gluing together code from existing libraries doesn’t count.

Liron 00:43:30
This one has been beaten so hard. This idea that AI can’t construct 10,000 lines of code — there’s just zero sense of this. I would even venture that the man himself would admit that he’s wrong on this one.

Ori 00:43:46
Are you catching that? Did you hear that? No?

Liron 00:43:50
Wait, did you do a sound effect? I didn’t hear it.

Ori 00:43:52
Oh, you didn’t? Oh, damn.

Liron 00:43:54
Too bad. Yeah, we gotta work on Ori’s sound effect setup. It’s okay. I’ll do my own sound effect setup here. “Just when I thought I was out, they pull me back in.” That one doesn’t make sense at that time. It doesn’t really make sense. We’re working. It’s a work in progress. We keep upgrading the show.

Shout out to Ori. If you’ve seen a couple of recent episodes, we are upgrading. I don’t know if you noticed we have better templates, the show’s looking better and better, so stay tuned. We got some upgrades in the pipeline.

Ori 00:44:21
Yeah.

Liron 00:44:21
But yeah, about Gary Marcus — 2029, right? And he predicted this. It’s not like these are ancient predictions. These predictions are now four years old. So it’s a four-year-old prediction that something three years from now isn’t going to happen, and it has already happened a few months ago. So that is concerning, as they say.

Liron 00:44:39
All right, number five of five. “In 2029, AI will not be able to take arbitrary proofs from the mathematical literature written in natural language and convert them into a symbolic form suitable for symbolic verification.” You know, Gary Marcus’s favorite buzzword is neurosymbolic. His whole point was, “These AIs, they’re just statistical, but they’re not neurosymbolic, and once they are, that’ll be a totally different world.”

Well, I’m pretty sure that they can take — I don’t know about arbitrary proofs — but they’re taking a lot of proofs, and they are in fact using a lot of formal methods on them. Whatever it is they’re doing, they are proving novel theorems. The Jacobian conjecture, I think, was the recent one to fall. We talked about the unit distance conjecture on the show.

Whatever he’s trying to say here — formalizing math proofs — technically maybe today I think he’s got a claim to say they can’t formalize everything right now. But humans also have trouble formalizing these proofs. If you tell a human, “Hey, take a proof and formalize it in Lean,” I think this is actually an active area of frontier development, our ability to formalize math. This is actually cutting-edge research. So saying that an AI can’t do it — yeah, I agree that an AI is not past the frontier of the smartest humans today in every respect.

Ori 00:45:50
Interesting. Okay.

Liron 00:45:53
This is shocking stuff that Gary Marcus wrote this stuff down confidently. And look, to his credit, I’m not looking at this and being like, “What a clown.” I honestly don’t think that. I think these predictions in 2022, I honestly would’ve disagreed with him at the time. Or actually, before ChatGPT, I probably would’ve only slightly disagreed with him.

I totally respect Gary Marcus writing these down. I think your opinion of him for writing these down should actually go up a little because of the fact that he had the balls to write it down and the fact that they didn’t seem that crazy at the time. So this is actually a point in favor of Gary Marcus.

However, by the time he came on my show in 2025, I feel like he should’ve already been hedging more than he did.

Ori 00:46:34
Yeah, yeah.

Liron 00:46:34
And actually, I feel like at the end of the day, he just seemed to be on the wrong track.

Ori 00:46:38
And also what his argument was is that if a system can do three of these things, AGI has been met.

Liron 00:46:46
Right. Exactly right.

Ori 00:46:49
So it’s the video —

Liron 00:46:49
Yeah, I mean, the frog’s getting boiled.

Ori 00:46:52
Well, it’s got the video one for sure, right? So the first one seems met. The second one seems met. The cook one — that one’s not quite. The fourth one seems met, bug-free code. And then the fifth one, I don’t know. I guess we’re not sufficient math experts to know exactly what’s going on in that space, but that also seems very close or met.

So yeah, should we be declaring that AGI has arrived?

Liron 00:47:20
Exactly, and the funny thing is he ends the post by saying to Elon Musk, “Deal? How about $100,000?” So I don’t think Elon Musk took him up on that, but otherwise he’d currently be in a lawsuit right now where Elon would insist on getting his money early.

Ori 00:47:34
Dude, that’s hilarious actually. I’m surprised Elon didn’t go legal on this. He’s pretty litigious these days.

Liron 00:47:44
All right. So that’s the Gary Marcus predictions. But it’s sobering because the extrapolation here is that everything we’re doing with our brain — our brain is going to be useless. And to me, the most visceral feeling of that is related to coding in my job.

I feel on a daily basis — I’m not gonna lie, it still feels good because I’m more productive. I’m issuing all these commands, and all this stuff is happening. But then in the back of my mind, not that far in the back, I’m thinking, “Okay, extrapolate this two months, and I’ll just give one command.” I’ll be like, “Look, what should I do? Go start your own threads.”

A lot of people are asking, “Who can start the thread? Who can kick off the initial prompts?” In two months from now, I’ll just be like, “Look, just give yourself a bunch of reasonable prompts.” I’ll literally just say that, and it will do it. And then I guess I will go to the park all day. I don’t know.

That’s what I mean by the visceral — that’s my day-to-day lived experience of AI, just the waterline doing everything better than our brains do it.

Ori 00:48:49
Wait, so your call is in a few months from now, you think it could just be doing everything?

Liron 00:48:55
Well, remember we met people from some unnamed AI company? We met them at some conference, and they were just telling us this. They’re like, “Yeah, agent swarms, they take it up to the next level. They run the company. They just do everything. Nobody’s gonna have a job.”

Ori 00:49:09
Yeah.

Liron 00:49:11
I can tell I’m on the ground here. I don’t have an agent swarm. Technically, I’m not at the very frontier of how people manage multiple agents, but I do manage multiple agents. I’ve gotten to the point where I literally have sometimes up to 10 different tabs of Claude that are all running, and I was kind of forced into this because Claude has gotten slow.

Slow in the sense that it’s doing a ton of work, but it’ll still work for 10 minutes, and it’ll do a crazily thorough job. It’ll do a lot of highly valuable thought process and operations during those 10 minutes. It’ll design tests. It’ll open a browser and click around and test. It’ll do all this fancy stuff, but it will take 10 minutes, so I can’t just wait for it.

So I go to the next Claude, and my brain right now is like spaghetti. It’s like a hash. Because I’ll look at whatever the last Claude said, and they alert me. They have a notification sound, so I go to whatever one notified me, and it’s like, “Okay, here’s two pages about the task I was doing,” and I’m like, “I don’t really need two pages. I just told you one sentence. Let me try to get the takeaway and write you a one-sentence response.”

Go to the next Claude, and this is a lot of context to hold in my head, but I’m just trying to do my best. It feels like spinning plates. So that’s my agent swarm. I’m manually going to all these threads and telling them a few instructions to continue. But my prediction is a couple months from now, I’ll just be like, “Hey, you know how there’s all these chats open? Just talk amongst yourselves. Just figure this out.”

Ori 00:50:27
Yeah. Damn.

Liron 00:50:31
Somebody’s saying Ori’s been frozen. That’s kinda weird. Yeah, I can see on the YouTube you look frozen. Ori, you might wanna reconnect to the stream.

Ori 00:50:40
Okay. I’ll hop out and hop back in.

Liron 00:50:42
All right. But am I frozen? Let me check. No, I seem to be moving. All right. Hold on. I think I forgot to blow the whistle. Minwoo Kim, $5 USD. He says, “Gary Marcus is hilarious.” Thanks. Appreciate it.

Liron 00:50:57
Let’s see. I’ll give some people some insider updates. I guess Ori’s not here. I’ll start giving you guys some insider updates. I’ve been doing some interesting episodes that are recorded. They’re in post-production right now. Producer Ori’s working on them.

Some of our upcoming episodes are with Michael Vassar, former president of the Machine Intelligence Research Institute. Former collaborator of Eliezer Yudkowsky, friend of Eliezer Yudkowsky. Interesting guy. I’ll leave it at that. He’s interesting. He’s multidimensional. He’s polarizing. So you might enjoy that episode.

And then we have a very interesting episode with a young gentleman. I would say young adult, but it’s hard to classify as an adult because he is 14 years old. His name’s Eli Goldfine. He has his own podcast. That episode’s coming out soon. We had a really good discussion. He absolutely is going toe-to-toe with an above average guest on my show, which is crazy.

I can’t help but joke about how he’s 14 — we’re saying, “Hey, is your P(doom) high enough that you’re expecting to experience high school?” Because he hasn’t even started high school yet. But he’s not gonna beat the youngest guest we’ve had on the show so far, my son Ezra. He’s seven years old. So Eli is ancient by Ezra’s standards.

Ori 00:52:42
Yeah. No, that joke hits a little — it kinda hits home for me actually. It makes me a little sad.

Liron 00:52:49
Yeah, you’re right, I shouldn’t joke about that. It’s a serious topic. But it’s gallows humor.

Ori 00:52:55
Yeah, for sure. I know, and I mean well, but yeah. When I think about younger people, they’re more innocent, and I’m like, “Oh, it’s sad.”

Liron 00:53:04
Yeah. Here, Eric Bierwa — I think I’m pronouncing that right, but correct me if I’m wrong. He’s saying, “I use 5.6,” talking about GPT 5.6 Sol, I think. And he says, “I use worktrees, like Git worktrees.” He says, “I can do parallel development on any number of coding tasks under different agents. I dispatch the agents from another agent and instruct the parent to handle any merge issues.”

Totally, yeah. Eric, I don’t know what harness you’re using or what front end, but I highly recommend conductor.build. It’s crazy good. It’s better than anything native. Tell me what you’re using. I’m curious. Maybe you’re using Cursor, maybe... What do people use these days to manage all their stuff? Maybe you’re using the new Slack integration.

I’m using Conductor. I’m really happy with it. They just released Conductor Cloud, which I’m really excited to try, and that’s what gives me the different tabs with the different agents running.

Just to give you guys some of the crazy capabilities — if you’re not programming at all — they have skills. I have a deploy skill, which is, “Hey, do everything it takes to take the code from my machine up to my web server.” Do all these checks, code review, make sure the site doesn’t crash, minify, bundle, do all this production stuff.

I tell my agents to do that, and it’s kind of elaborate. How do you deploy different things at the same time, like Eric mentioned? But I don’t write the Claude skill. I tell Claude to write the Claude skill. I’m like, “Hey, improve your skill to be able to do this.” And it’s like, “Okay, here’s four paragraphs of instructions to myself to do this better.”

They have a memory feature now, so it has context. It’s like, “Hey, I remember we previously worked on this, and I taught myself some lessons about mistakes I’ve made.” It’s just crazy. It’s a crazy time.

Liron 00:54:50
Another way to describe it is this. I’m a one-person engineering team. My company is not really blocked on engineering. We’re a coaching company. We help people connect to coaches. So it’s not like we sell software. The team uses internal software, and we have a lot of software, but we’re not a software company. I’m the only software engineer.

But the way my experience has been with Claude Code the last few months, honestly, it’s about half an employee. It’s as if half a human employee, unassisted by AI — so there’s this metric, 2025 equivalent humans. Freeze a professional, let’s say $300,000 a year compensation. Freeze a $300,000 a year compensated human from the middle of 2025 with little to no AI assistance. Take a person like that. That’ll be the unit of measurement.

I claim half of those humans, so about $150,000 a year worth of human salary, is joining my company every week. So it’s been about four months now. My current experience is that it’s like me and a team of more than six. Twelve months, that’s like six humans. It’s like me and a team of at least six humans. And sure enough, I am in fact actually more than six times more productive.

So I think I’m actually underestimating the rate of humans joining my team because of all this.

Ori 00:56:07
Interesting. Yeah.

Liron 00:56:07
It’s a surreal number. These numbers, there’s no comparison to anything that’s ever happened in the history of business productivity that matches my experience of having an almost free high-level human employee essentially joining the team constantly.

Ori 00:56:23
Wow. Wow.

Liron 00:56:24
The funny thing is I had a company meeting yesterday, and the company meetings, most of my team is just the coaches. We’re basically a coaching marketplace. So me talking to the team, it’s just me talking to a bunch of coaches. I don’t think they watch Doom Debates. The percentage of Doom Debates watch hours from people who work at my company is very low right now. They actually prefer Warning Shots ‘cause it’s more accessible. Maybe they prefer these live streams. Shout out to Relationship Hero coaches if you’re watching this.

But anyway, I did the company meeting, and I’m like, “Yeah, let me tell you guys the business update. A lot of what’s been happening in the business is — if you’ve noticed, when you guys have suggestions for features, you know how that used to happen 30% of the time we try to accommodate you? I don’t know if you’ve noticed, but now it happens 99% of the time, within two hours of you posting the suggestion. All the suggestions are just getting done. We’re not limited at all by the bandwidth of getting your suggestions shipped out.”

“I don’t know if you’ve noticed that. Yeah, so some context about that — we’re currently in an intelligence explosion, and we have the equivalent of 12 humans working on the software team. And also, one of the risks of the business is that AI kills everybody.” I was pretty straightforward with them. “Yeah, I know you guys don’t think about AI much, but I just have to tell you this context of what’s happening to our coaching company.”

Ori 00:57:33
Wow. That’s incredible. Wow, that you told them that. I mean, I guess it is a business risk, isn’t it?

Liron 00:57:41
Yeah. It’s just funny because I think other companies, maybe they don’t have anybody at the company that’s following this close to the frontier. And again, I’m not personally claiming to be at the frontier. I see myself as being a few weeks behind the real astronauts. I’m not that. I’m a few weeks behind. I think being a few weeks behind, I’m already riding.

I’m in one of the middle or back seats of a roller coaster, but I’m still on the roller coaster. So it feels the same.

Ori 00:58:09
Yeah, for sure.

Liron 00:58:10
Eric Bierwa says, “Implementing code is so cheap now that the bottleneck has moved into decision-making.” Yeah, correct. “It’s fast to iterate with ideas by deploying them and tweaking what doesn’t work versus spending multiple weeks.”

Yeah, you gotta tell me, what front end are you using? Are you pulling up the Claude desktop app? You mentioned GPT. Are you pulling up the Codex Mac app? Specifically, what are you using? I’m curious. There’s also a good chance that your answer might be something I should go use, even though I feel pretty good about Conductor.

But yeah, when you mention all you do is make decisions — it’s literally this. I open up a new tab and I just say something like, “Hey, can we have it give the coaches a phone call when somebody schedules an appointment with them? Thanks.” That’s it. Ship it off, go to the park. Come back. It gives the coach a phone call when somebody schedules an appointment with them.

Ori 00:58:59
What deep disillusion—

Liron 00:59:00
Oh, okay. Eric, if you’re using the Codex desktop app and you’re using worktrees, I suspect you might wanna try Conductor, because I think Conductor is kind of built from the ground up for the power user who uses worktrees, and I think Codex might be a little bit behind in terms of what workflow they’re expecting.

All right. DavidPatten1 is saying, “When will it be able to take my family photos and sheet music of our favorite songs and a detailed description of our life over the last 20 years and have it output a playable game?” I think it can do that today. You literally just need to connect Claude Code, open up a session with Claude Code or Codex and be like, “Hey, here’s the API key to my Google Drive,” or “Ask me whatever you need. This is what I wanna do. Ask me what credentials you need. I’ll go grab them for you if you need me to.” And then take the next 90 minutes and do this together. If you honestly want that, David, go try it. As somebody who roughly knows where AI stands today, that is within the realm of what AI can get for you today.

Ori 01:00:01
Yeah. It could do that for sure. I would agree with that. It could make a trivial game, sort of like a quiz game, that’s kind of boring. But I wonder what kind of... what it could do that’s more creative, and I bet it could do some more creative things.

Liron 01:00:12
Right. And we can acknowledge — this is like the AI fan club. I’m excited about these superpowers. I like just saying what I want and then not spending time, and then having it happen for a pretty cheap price. I find that incredibly exciting, empowering.

It’s just, as you guys know, in case it’s not clear, I’m just expecting that the trend continues into rogue, uncontrollable AI. I just wish it wasn’t about to go rogue and uncontrollable.

Ori 01:00:39
Yeah. Speaking of which, I feel like we should mention the donation pitch for the show again, so that we could prevent AI from going rogue and uncontrollable.

Liron 01:00:52
That’s right. The middle of the episode, this is the sweet spot for when people are paying attention. You want to hit beginning, middle, and end.

Liron 01:01:02
Yeah, so just to reiterate, the show is a significant cost to run. We’re roughly $200,000 a year. We can cut to the bone lower than that if need be. But Ori lives in San Francisco, high cost of living area. But he’s networking. He literally does physically network with people. He’s getting his money’s worth for San Francisco, for sure.

Ori 01:01:20
That’s true.

Liron 01:01:21
I’m economizing. I live in Saratoga Springs, New York. But also, I want to drop this factoid, because I do not make income from Doom Debates. I am actually more like — I’m on your guys’ side of the table. I’m donating. I’m giving to the show. I’m giving my time. I’m giving my money. And when the bank account’s running dry, I’m giving my own money to the show. So I know what it’s like to be a donor.

Just to reiterate, I actually have a commitment for the next 12 months. I don’t want to commit longer than that — things could change. But for the next 12 months, I will continue, as in the last 24 months, to not take a single penny out of your donations toward my personal income. Why? Because I do currently work my other job. I run a company, and that gives me a paycheck, and I intend to just keep living off of that income and not on Doom Debates at all.

I intend to keep donating to Doom Debates. So I really hope that you guys do too, because we are facing a potential risk scenario right now where I can’t really pull the whole production cost from my personal savings for more than another month or two if literally nothing comes in. I don’t think that’ll be the case, but it is becoming a bit of a struggle.

The worst-case scenario is we have to start cutting. We have to be like, “Okay, let’s do fewer episodes.” Maybe Producer Ori has to get another job. These are nightmare scenarios. I hate that we even have to talk about this, but we did kind of let the money run dry to the point where this really is becoming a possibility.

But it doesn’t have to be. We could be talking to you guys two weeks from now being like, “Hey, a few people donated a few thousand dollars, and we’re feeling pretty good.” And as you know, we do have some milestones — even before the Anthropic IPO, for example, when we’re expecting a lot of people to get very rich, and we’re expecting money constraints to be less. A lot of organizations like us are probably going to be in some kind of queue to get a sizable donation. So that’ll kind of solve that problem, I think, pretty likely.

And even before then, there’s Lightcone Commons, which we feel pretty good about. In about three months, hopefully getting tens of thousands of dollars as a grant from Lightcone Commons. Fingers crossed, no guarantees.

But we’re also just thinking about short-term. Right now today, we’re pretty short on cash. And we’re trying to be realistic about how we’re gonna plan — are we gonna do the same production quality? So that is the honest, transparent assessment of where we are right now.

For more context, the show, until today, it’s a remarkable fact that the show has been 100% viewer funded. In the first year it was just 100% me working on my own time and paying expenses — I bought myself a video camera. So it was basically zero funding. And then the second year, surprisingly generous viewer donations. We made it through the whole second year. And then we’ve just been a little low on viewer donations recently. That’s why we’re doing a donation push.

But we’re optimistic that if we can just recapture some previous level of viewer donations — meaning multiple donations of multiple thousand dollars — then we feel really good about the show. Anything to add to that, Ori?

Ori 01:04:20
Yeah. You said that really well. And also, the one thing I was thinking too is that I feel like if big enough funding came through, then I feel like you could work on it full-time also.

Liron 01:04:32
Right. I don’t — I didn’t wanna dare suggest that. I don’t wanna set my sights too high here in terms of what’s possible. But yeah, you know what? Sure. I’m gonna go there.

Look, zoom out. What does the world need right now? If we’re being really ambitious, what does the world need? It probably needs me, Liron Shapira, to focus full-time on Doom Debates. That probably is what the world needs.

Is it what my wife and kids need right now given that I have a pretty high cost of living? Maybe not. So I’m trying to have it both ways right now. But hypothetically — and look, I’ve grown accustomed to an engineer-level lifestyle. As you guys know, engineers get paid a lot. So it’s a big sacrifice — I would literally have to move into a smaller house. That’s not the worst problem ever, but it’s tough.

So hypothetically, if money weren’t an issue, if somebody wanted to give me an engineer-level salary, if some billionaire wanted to do that, there is an argument to be made that that’s good for the world. But I’m not here pitching that to you guys right now. I’m pitching that I’m committed to not taking... I guess there’s a hypothetical situation — if some donor was giving us money and said, “I insist that Liron take this money,” I guess then I should violate my vow to never take money. If that’s what the person wants. But that’s not the default outcome.

Ori 01:05:51
Yeah, for sure. I think you have to support your family, obviously. You have to put that first. But if you could do that and do this show full-time, then you should. If that comes through, if there is the support for that, then I think you should do that.

And I could say for you — maybe that sounds a little more reasonable — that yes, I 100% agree. That’s why I’m working on the show. What does the world need right now? We need more Liron out there. We need more Liron holding the wishful thinkers accountable, talking about the risk, because otherwise we’re just gonna get... We’re playing a small part in the AI safety ecosystem.

I wouldn’t be able to say, “Oh, you could talk to the president and convince the president of something.” But we’re a small part of the ecosystem, and we wanna just be a bigger and bigger part. So that way more people can be made aware of the risks. I think it’s really important work what we’re doing. We gotta get you out there more, for sure.

Liron 01:07:01
Hell yeah.

Liron 01:07:05
Yeah, there is an opportunity to take the show to the next level. There’s a couple of dials that we can turn if we had more than our baseline budget. Our baseline budget gets us a lot. It gets you continued three episodes a week, continued pipeline of increasingly prominent guests, holding people accountable. Go see our donation video, our state of the show video.

But there is also a dial we can turn. We haven’t really planned for a significant sized marketing budget. We could do marketing — we’ve got the content, let’s market the content. So that’s also something we could spend on. And then as Ori mentioned, we could spend on getting me full-time.

There are episodes I wanna do. There are canonical reference episodes I wanna do. I still haven’t gotten around to doing the reference episode arguing against every single stop on the doom trip. I haven’t done the reference episode about why Sam Altman has no idea what he’s talking about when it comes to mitigating AI risk. I’ve compiled enough Sam Altman statements. So it’s on my to-do list to do an episode like that.

The show is constrained like that. That said, the highest marginal impact is from just at least continuing the momentum that we already have. This is the worst time to slash the momentum — the same week that Hugging Face is getting hacked and these AIs are going rogue.

Ori 01:08:15
Yeah.

Liron 01:08:15
One more thing worth mentioning. I don’t think that this is a crowded space of shows with a host that has an opinion and is confronting people about that opinion and having high-quality discussions about that opinion. I think you’re going to find a lot of interview shows, a lot of friendly, sycophantic interview shows.

I don’t think you’re going to find a lot of critical forums and productive forums where ideas clash and mix in a mainstream accessible way. I think you’re looking at a pretty small set of competitors here. And if we were to turn the lights off — which I don’t think we are, we’re not gonna turn the lights off — but if we were to cut the quality of the show, I just think we would be leaving a vacuum for this kind of show.

Ori 01:09:04
Yeah. That’s 100% the case. And also, by the way, Liron, your screen share is still showing, and we’ve just been talking for a while, so it’s probably —

Liron 01:09:15
All right, one sec. Yeah. All right, thanks for the heads-up.

Liron 01:09:19
Okay, hold on. We got some donations coming in. $5 US from David Patton, and he’s saying, “What is on your pre-extinction bucket list?” Good question — let’s put a pin in that for a sec.

And then EJJ2025 with another $4.99. He’s saying, “Why are you so confident that we will get a rogue ASI soon? There seem to be many limitations to get ASI.” Okay, let’s finish out the donation arc. David Patton’s also saying, “What is the best way to fund the show?” These are such great questions. We appreciate the question.

The best way to fund the show is to head over to doomdebates.com/donate. I’m gonna go get busy with this feature here on the live stream. It lets me post text. It’s doomdebates.com/donate, and then it goes through our Manifund page. If you go there, it’ll take you to our Manifund page.

Manifund is actually our 501(c)(3) charitable sponsor. So if you are wealthy and you want to make a charitable donation, you can totally get a tax deduction for doing that, and it goes right to Doom Debates.

Ori 01:10:13
Yeah. And I wonder — to what degree, if you want to help AI safety, where’s the best place to put your money? I don’t know. Bang for the buck, I think it’s pretty good money on Doom Debates because it’s very unique. Who else is having these kinds of conversations?

It’s a form of investigative journalism really. That’s part of it — to bring someone in who’s supposedly an expert, and then suddenly you find out there’s nothing there. There’s a wizard behind this or something. The safety arguments that you thought they had are actually exposed to be very, very weak.

Liron 01:11:03
I think Ori and I both have a fetish for emperor has no clothes situations, right, Ori?

Ori 01:11:11
I think you do, and I think maybe you’ve —

Liron 01:11:15
Okay, I do. I’ll give into that.

Ori 01:11:16
You do, and I think maybe you’ve turned me onto it. It’s so shocking and absurd when you see it. So you really gravitate towards that. For me, I’m kinda like, “Yeah, okay.”

Liron 01:11:29
I definitely gravitate to emperor has no clothes situations.

Ori 01:11:34
Yeah.

Liron 01:11:34
And look, Sam Altman is an example. Not to pick on him personally — all these AI leaders. But I consider him the emperor who has no clothes. He is in the place of trying to reassure people that it’s fine, like the “this is fine” dog, when it’s clearly not fine. It’s clearly completely messed up what he’s doing.

Ori 01:11:50
Yeah, for sure. And same thing with all the AI safety CEOs. Dario Amodei, Demis Hassabis, Elon Musk, Mark Zuckerberg. They’re all in one way or another looking at the risk and being like, “Eh, whatever.”

Liron 01:12:07
Yeah. All right. 10,000 units of currency in HUF from AdamRack7560. He says, “You should get well-reasoned red lines from anybody who has near-zero P(doom) and officially record them.” Yeah, basically the Gary Marcus strategy. “Basically, what are the very well-defined checkable red lines where you start worrying?”

That should be a standard question on the show. Maybe I haven’t been super rigorous about asking that. Sometimes people ask me that. They’re like, “What are your red lines, Liron? Everybody’s still safe today, aren’t they?” And I always say, “Well, wait till AI is superhuman at achieving outcomes, which it’s getting closer and closer to.”

That’s my line for not being worried. AI can superhumanly achieve outcomes, and then somehow the equilibrium of not having these superhuman wish-granting genies roaming the earth, somehow that equilibrium persists for a little while and everything’s fine. At that point, I question my understanding of intellidynamics — the consequences of having high intelligence operating in a universe.

That seems to contradict my understanding of intellidynamics. So then I rethink the theory. But it just seems unlikely that you just have these superhuman forces doing superhuman stuff, but it’s fine.

Ori 01:13:25
Yeah. All right. Minwoo Kim said, “Wasn’t there a billionaire who committed 50 million to prevent AI doom, and Roman Yampolskiy decides which projects receive the funding?” He might be talking about the Lightcone project that we applied to.

Liron 01:13:45
Yeah, it could very well be. You guys can go to lightconecommons.com to see what we’re talking about.

Ori 01:13:52
I don’t think he’s on that list, but it just sounds very similar to that, so maybe he’s referring to that.

Liron 01:13:59
Yeah. They might be talking about Survival and Flourishing Fund. In transparency, we did actually apply to that a few months ago, and we had a lot of support from the people in Survival and Flourishing Fund. Multiple judges were like, “Yep, I want to fund this.”

But in the end, there were some issues. They pulled back from funding advocacy of any kind. I actually heard a bunch of stories of people who were doing advocacy like us. And there were some differences in opinion from some of the leadership of Survival and Flourishing Fund with some of the way that we approach communication. We communicate strongly. Sometimes we communicate edgy.

And to be fair to them, I’m talking a little vague now, but to be fair to them, I even communicated in a way that I would’ve tweaked in retrospect. But anyway, long story short, Survival and Flourishing Fund did not work out for many of us in the AI communication space. And one of the reasons why many people are excited about Lightcone Commons is they’re doing certain things differently. My understanding is they don’t have similar views about funding communication like ours.

Ori 01:14:59
Nice. That’s awesome.

Liron 01:15:01
But it’s in three months from now. We’re in a tough position where we’re taking a risk in terms of how much we wanna invest. Because we just don’t have odds for this kind of funding. And I personally have never received a grant of significant size from any organization in my life, unless you count venture capitalists. I do have significant experience with venture capitalists giving me more money than I deserved. But that was a different part of my life.

Ori 01:15:27
Yeah. But you’ve learned a lot from that.

Liron 01:15:30
Exactly. All right. There’s a segue — we were talking about emperor has no clothes. We can segue to talking about the whole kerfuffle that happened earlier this week with Seb Krier, Google DeepMind.

Ori 01:15:44
Dude, that is exactly what I was thinking. Talk about communications and communications nuance — wow. Talk about not having that kind of nuance for someone in a key policy leader position at Google.

Liron 01:15:58
There are two connections you can make to what we’ve been talking about. Number one is what you’re saying — talking about standards of communication. We’ve been criticized for some of our harsh communications. Criticizing people, talking about how there’s a high risk, and there’s — we should call for potentially having policies that lead to enforcement using weapons. Weapons of enforcement, airstrikes on data centers. We’ve been criticized for all kinds of quote unquote “violent rhetoric” like that, which I completely disagree is violent.

And we’ve been criticized for saying — Holly came on the show, and she’s like, “Holly’s basilisk” — these people who are currently saying we’re just following orders to build AI, or somebody else is gonna build it. Well, that’s kind of like saying, “I was just following orders,” in the Holocaust. And I argue with her. I’m like, “Look, but don’t you think that if it’s worth fighting, then we should just make the law right now? We shouldn’t retroactively prosecute them under some laws?” And she’s like, “I don’t know.” So we slightly disagreed on that issue.

But anyway, we’ve been criticized. People like me who are doomers get criticized about bringing this kind of stuff up. And so it was very interesting that the tables kind of turned a few days ago, if you guys watched the episode — Monday’s special report about the Seb Krier shitpost.

He shitposted the meme image. It was him — it was this meme where he’s a character holding a rifle, and then there’s people coming through his door who are wearing hats that say PauseAI and LessWrong. If you know the meme, one of them is a schizo — I don’t even know the meme. But whatever it is, he went very edgy. He adopted a kind of edgy community style, the kind of edgy style that you might find on the other side of the table. He very much adopted it for his side, but I didn’t think that was right.

The tables kind of turned where I’m like, “I don’t like that you’re doing this. I think you’re actually crossing the line, and you’re crossing it more than I would personally feel comfortable crossing it right now.”

Ori 01:17:48
Yeah, totally. There’s a lot of defense on Twitter, which is where this all went down. And for listeners of the show too, Twitter’s an important place for the discourse because it is where the AI discourse goes down. The norms and expectations of what people say on Twitter, it is influential because that’s just where influential people are talking and following things.

Liron 01:18:20
Let me just give people a little bit more backstory. So he’s the lead of frontier AI policy at Google DeepMind. When I think about who at DeepMind is responsible for their policy, I think of Demis Hassabis, the CEO of Google DeepMind, and obviously Sundar Pichai, the CEO of Google generally, and I think of Seb Krier, because his title is frontier policy at Google DeepMind.

Well, conveniently, Seb Krier seems to tweet a lot, and he seems to tweet in this very authentic style, so you really get to know what he’s thinking. And what he was thinking one day last weekend was that he’s so annoyed by Pause people that he wants to tweet a meme saying that people harassing him to pause are so troublesome for him.

But again, as we said in the episode, the problem with the meme is it does depict him modeling this image that you can point a rifle at them. There was literally a rifle pointed at the face of the person with the PauseAI hat. And the obvious defense is, well, if you’ve seen the meme, you know that that’s where the rifle points.

The thrust of my episode was, unfortunately, when you are frontier policy lead at Google DeepMind, the defense that you have to use the meme familiarity lens in order to judge your communication — that’s just not how it works. That’s not how public communication works. Anybody could see that tweet. He’s the frontier policy lead at Google DeepMind. People who aren’t mentally stable can see the tweet. People who aren’t terminally online and know memes can see the tweet. You just can’t tweet the depiction of your character pointing the gun at the PauseAI people.

Ori 01:19:51
Can we put it on the screen, actually? Do you wanna do screen sharing?

Liron 01:19:55
Yeah, yeah. Just one sec. David Patton with a $10 donation. He says, “I just subscribed to your Substack. Do you watch War Games? Can you talk about Arcade GI3?” All right. Noted with the request.

Let’s go put the tweet on the screen. Or maybe, the easiest way to pull it up since he actually blocked me and took his account private — yeah, it’s my fault for making a ruckus out of it, so I’m the blocked one here, guys.

Ori 01:20:24
I think it was a screenshot on Twitter that you posted.

Liron 01:20:28
Yeah, exactly. So the easiest way for me to show it to you is to just show my screenshot that I posted, so I’m just scrolling down toward that. Here we go.

Ori 01:20:37
The concerning tweet.

Liron 01:20:40
Yeah. Here, I got it. Can you see it now?

Ori 01:20:43
Yes. It’s weird how I see that shirt. Yeah, so the context —

Liron 01:20:49
I do encourage you guys to watch the Doom Debates episode from Monday night. So Roon from OpenAI was saying — the context of all this, which I said in the episode, is there’s been a lot of discussion about pausing AI, and this was also right before the PACE letter came out. So this is literally two days before the PACE AI letter comes out, or I guess four days when this happened.

There’s this groundswell of pausing AI. Demis Hassabis said on stage, “Hey, I’m open to pausing AI. I think that’s probably the way to go if we can coordinate it.” And Roon tweets, “If we could coordinate a global capabilities slowdown today, I would likely press that magic button.”

So this groundswell is happening, and then the frontier policy lead of Google DeepMind takes a dump on that idea. He says, “My face when” — basically my face when people are bringing up PauseAI. You can see in the image, it’s people with PauseAI.

Ori 01:21:36
Can you zoom a little bit? Like, zoom on the character and the fact that he’s holding it, maybe?

Liron 01:21:41
Yeah, sure. Give me a sec. The zoom — I know a way to zoom. Give me a second here. Here we go.

Ori 01:21:49
Nice.

Liron 01:21:49
So you can see — this is the salient part of the image. I’m just following the rifle barrel, and it’s going straight to the head of the individual from PauseAI saying, “We must pause.” And there’s a LessWrong person saying, “We must pause.”

So that’s what he tweeted. And as I said in the episode, there’s two problems here. The first problem is that I thought Google DeepMind was on board with pausing, so I didn’t realize that their frontier AI policy lead is lightheartedly saying that people harassing him to pause are so troublesome for him.

And of course, the other troubling thing is that you just can’t be the frontier AI policy lead, a leadership position at Google DeepMind, a $4 trillion company, and put out this content. So for those two reasons, it is, as I said, concerning.

Ori 01:22:44
Yeah. Well, let’s talk about both points. One — about it conflicting with where Google stands on it. It’s not just that he’s making a lighthearted joke about this pause or slow down capabilities policy. What is the meaning of the joke? The literal meaning of the joke is that I am so against this policy that I’m gonna hold a gun. I’m gonna be the last man standing, holding a gun.

There’s the famous Charlton Heston quote where he’s like, “You can take my gun from my cold, dead hands.” He’s like, “You wanna pause AI? I wanna be in my room doing my compute work. You gotta take it from my — come and get it.”

So that’s one thing. It’s not just that he’s making a lighthearted joke. He’s gonna be like, “I’m going down in flames. I am not giving this up at all.” That’s the meaning.

Liron 01:23:45
Right.

Ori 01:23:46
So that’s one point.

Liron 01:23:47
Exactly. I will die on this hill of saying that PauseAI is so annoying. It’s very important to take a stand against PauseAI — coming in the wake of Demis Hassabis saying, “Yeah, pausing sounds like the right way to go.”

Ori 01:24:00
Yeah. So that’s one point. And the second point is the violent imagery, the violent meaning or connotation that could be inferred from it. And this is what people on Twitter got so upset about. They were like, “Come on, man. It’s just a meme. It doesn’t actually depict violence.”

It was a whole conversation. People were really talking about this a lot. A lot of people weighed in. And I don’t know about you, but I got a response from a dude who said something to the effect of, “We’re not gonna pause. Pausing is not an option.” And then he posted a photo of a literal gun.

So people are like, “Oh, this is just a joke.” But okay, someone seems to have oddly made it a little more literal. How far away is it from this comic to actual real life? It’s not that far removed to get to that in real life.

So I think doing that image is unacceptable behavior, for one, to even depict that for someone in his position. If you’re making jokes, if you’re being edgy, then nuance is required. And in this case, I think the nuance stepped over the line because you have a gun pointing at a real figure. In most cases, this is a meme — you kind of put things to absurdities. But I think he made some kind of communications mistake where there are too many literal elements here, and it just goes too far.

Liron 01:25:56
Now get this — you may not realize this, but the second chapter to the story is there was a huge backlash to me daring to point this out.

Ori 01:26:08
Yes. Yeah, huge backlash. People are like, “Wow, all these PauseAI people, this is such a self-own. You guys are wasting so much time. You guys are idiots.”

Liron 01:26:17
Yeah. I just checked Seb’s account here. It’s still in protected tweet mode. So I think the result of this whole kerfuffle is now he’s left the tweet up. The last time somebody sent me a screenshot of his private account, he left the tweet up, but he’s like, “You know what? It’s so important to me to be freely putting this tweet up that I won’t have it public, so you can’t accuse me for publicly posting this, but I’ll leave it up.” He’s dying on this hill that this tweet needs to be up.

Ori 01:26:43
Unbelievable. Unbelievable.

Liron 01:26:45
But let’s talk about the backlash, because the backlash — I don’t think I personally was the number one thing that instigated the backlash. I definitely pissed a lot of people off, and I got my own backlash, but I think what really took the backlash into the stratosphere was a number of other PauseAI people, like Holly and Maxime — don’t know how to pronounce his last name, I think PauseAI France, if I understand correctly.

Ori 01:27:12
He’s the head of PauseAI now.

Liron 01:27:13
The head of all PauseAI. Okay, great. Sorry, my bad. So these people were like — and I totally respect this — they’re like, “Hey, I see this as threatening. I think this increases the threat to me.” And I think they also went so far as to be like, “I don’t get the meme at all. I just literally see this as a call for violence.”

I personally am okay with them saying that, because there’s no requirement that somebody looks at a meme and gets it the way you think your in-group is going to get it. There are standards of global communication. When you are the frontier AI policy lead at Google DeepMind, your tweet is a global communication. It is not just you and your homies. It is not your DM group.

If somebody caught Seb doing that in the DM group, then it’s like, okay, yeah, sure, that sucks that he’s doing that, but he doesn’t need to be fired for that.

Ori 01:27:58
Right.

Liron 01:28:00
So Holly and Maxime and a number of other people are like, “I see this as threatening.” And by the way, I will also raise my hand and say I see this as somewhat threatening. I see the existence of these memes going unchecked, this imagery going unchecked. Every time somebody does it, it’s normalizing more and more. Like, yeah, I’m comfortable with this imagery. This imagery is okay. And you really shouldn’t be comfortable with it.

That is why it’s a zero tolerance policy. I don’t want Google DeepMind’s high-level representative tweeting this. I’m not comfortable with that. Am I losing sleep? Not really. It’s not quite at the level where I’m like, “Okay, I need to hire a private security guard,” thankfully. It’s not at that level. But does it turn the dial toward there? Yes. Yes, it absolutely does.

And Holly’s somebody who’s even more prominent and she’s more on the ground — and she’s also a woman. I feel like people go after women. Women are more easily victimized. They’re the weaker sex. Crazy people lash out at women, I feel like, more than men. So I completely empathize with her being like, “This is threatening to me.”

But what happened — I gotta tell you guys what happened. You don’t even realize what happened on X a few days ago when this all went down. What happened was there was a massive backlash inside the people who normally comment on AI, the kind of people that I read. People from a bunch of organizations, AI safety organizations, people from a bunch of AI companies. Roon himself was part of this crowd.

A lot of people came up and they’re like, “You guys are being so lame. You’ve completely discredited yourself. You PauseAI people. The fact that you would play the victim card.” They basically treated us like — imagine a soccer game and somebody’s pretending to foul, lying on the ground, having the ref come over and he’s totally faking it ‘cause he wants to play the injury card.

That’s what they thought we were doing. They thought we were faking this injury just to get the ref to come over and punish Seb, where it’s like, no, the communication needs to come down. Why is he leaving it up? All he has to do is tweet it again in a style that’s more acceptable. It’s not that hard. It’s not the craziest ask.

Really what Seb should have done — the correct move is to be like, “Hey, I tweeted this out. I was joking. I didn’t mean any harm to anybody,” which is what I presume he’s thinking. I don’t have a reason to think that he’s a bad person — well, besides working at an AI company. But regardless of what he’s thinking, the correct move is to just tweet out, “Hey, it was not my intention to have any doubt that I’m not calling for violence.”

There is a sense in which such communications do have too much doubt. It is in fact too much doubt according to what is considered the standard of communication. “So I messed up and I won’t do it again, so I’ve taken it down.”

Ori 01:30:40
100%. And honestly, it is shocking. It is shocking that this image is still up. Think about what it signifies. The policy leader of Google DeepMind is okay with a depiction of violence against PauseAI. And that is in contrast to what his CEO says.

Many people — if you look at it, there was just this big letter that came out about pacing the frontier.

Ori 01:31:11
Many people in the AI companies are saying, “We need to have an option to slow down.” And people who signed it are top, top leadership at Google DeepMind. You had the co-founder of Google DeepMind, Shane Legg, signing it. The head of alignment at Google DeepMind, Anca Dragan.

There’s other top leadership at Google DeepMind who signed this letter, and the policy leader is posting, “This is such a bad idea. I’m gonna go—” his sentiment, his feeling is, “I wanna be this schizo holding a gun with it.”

Liron 01:31:50
Yeah, exactly. No, it is shocking stuff. Now we have a microcosm of debate. Our own viewer, Punmaster STP, he’s giving a representative sample right now in the comments of the side that we are arguing with on Twitter.

He’s saying, “If I or someone else said that they found the endings of the old Warning Shots episodes threatening, would you, John, and Michael really take them down?” So Punmaster is saying, look, at the end of Warning Shots, which we stopped doing by the way, but if you go back a few months, me and Michael and John at the end of Warning Shots would be like, “All right, this was an episode of Warning Shots. Everybody fire your guns.” And we’d go like this with our fingers, and we’d have gun sound effects, sometimes visual effects. “Yeah, warning shots, everybody. We got guns.” Okay? Just to make things fun.

So if somebody—he’s trying to make me empathize—what if somebody complained, “Oh my God, Liron’s firing these guns in the air. He wants to go assault AI companies.” So first of all, just to remove any doubt whatsoever, we have stopped doing that. Even though this is a very edge case, I feel like this is a very slight case of this. This is not as bad as posting a meme where—don’t forget the person at the end of the gun.

Follow the end of the gun. In my example right now, you can see that I’m shooting the gun into the air as a warning shot. The show is called Warning Shots. The warning shots are about AI being a warning shot. The abstraction here is very clear. If you look at this image, is there really a lot of abstraction when the barrel of the gun is held by the sub character and pointed at or toward somebody wearing a Pause AI helmet?

Do you think that there’s maybe a little bit less abstraction in that particular depiction?

Ori 01:33:20
Yeah, for sure. And so what is the message that Google DeepMind is saying? What is the message about their policy? That it is okay to—you know, “We think that a capability slowdown is important.” People inside of Google, people who are top senior leadership inside of Google think that’s important. And the message from someone who’s head of policy is, “That idea is so dumb,” and, “I would rather go down with the gun.”

Liron 01:33:53
You know, I gained respect for Holly with this whole interaction because you know how she’s always saying that Anthropic is kind of the kingpin and all of these effective altruists are the insiders in the community. They’re all this echo chamber where they reinforce each other and they have this frame that they’re the insiders. They know what’s up. They can handle this. They’re like Sam Altman. They can manage through this fine and it’s all about nuanced debate and we’ll have a conference with each other.

And of course pausing AI now is too extreme but we got to find a middle road. Whereas meanwhile Pause AI people are doing these lame protests and just yelling, “Hey, you guys are creeps.” And they’re like, “No, we’re not creeps. We’re cool. We get it.” And there’s people at the AI companies who will come and talk to us and be like, “Look, Dario seems to get it. Dario’s a very thoughtful guy and we’ve had conversations. We’re gonna do something when things get too crazy. We’re gonna take action ‘cause we’re gonna be at the front of the trains. We’re in a good, we’re in the catbird seat. We can stop this.”

But it’s like, no, you’ve got an over-inflated sense of your power. This is all—you’re a useful tool basically. You’re the AI’s tool essentially right now. It’s not going to work out well. Just because you work at an AI company you’re not gonna get spared by the AI. So there’s this naive thinking and it was on such full display when Seb did this tweet and everybody, even people that I have the utmost respect for, they would chime in.

I’ll name names. Let’s say Steven Byrnes. He’s a top A+ level respected guy in my mind.

Ori 01:35:20
Oh, Steven Byrnes. I don’t have—

Liron 01:35:21
Okay. Yeah, Steven Byrnes. I just want to cite him because he even chimed in and he was like, “Look, do you really think that it would be good for Google to fire somebody for a tweet like this?”

And by the way my position on whether he should be fired is I think it was a fireable offense, so if I hear the news that Seb got fired because this tweet was inappropriate I’d be like, yeah, that checks out. I’m not surprised that he would get fired for something like this. But if his boss talked to him and was like, “Hey, you got to take this down and never do this again. Can we talk about why you did this? Can we talk about standards? And this is a first offense, so don’t do it again.” If that’s what happened and he didn’t get fired I personally would also be okay with that. I’d be like, okay, good enough.

I think Holly wouldn’t be okay with that. I think Holly’s like, nope, this is unforgivable. And it’s like, okay, so we can have—there’s a little bit of a gap in our opinion there. But then Steven Byrnes was on a different page. He was like, “I don’t think people should get fired for posting this. It’s useful to me that when Seb is on his own time, when he’s not at work, when he’s just letting us know what’s on his mind, it’s useful that he can be honest, and it would really be silencing him if he would get fired for that.”

And okay, but generally, it is a fireable offense. I’m just kind of telling you what it is. And if I could architect the policy of a frontier AI company, would I want it to not be fireable? I don’t know. I think I’d still want it to be fireable. But I guess reasonable people would disagree. But why are you kind of coming at it from this angle?

So Steven Byrnes, he’s not—I don’t have a beef with him. I respect him. We had a good back and forth. But it is representative of how this really sucked in everybody who’s anywhere near the conversation. It literally felt like me, you, Holly, and a couple others were these lone voices.

And there was even a handful of people who were like, “I’ve unfollowed you for this. This is completely discrediting.” I guess I’ll call out Jack, Tracing Woodgrains. Somebody I think—

Ori 01:37:04
Oh, yeah.

Liron 01:37:05
I think we’re still mutuals. I like his work. But he was like, “This is so discrediting. You’re just kicking up dust. You’re just opportunistically manufacturing a controversy where there’s nothing here.” Everybody felt very deeply in their hearts, “This is our community, okay? When we post memes to each other, it’s because we get each other. We’re not in doubt about the interpretation of what people mean when they post images like this.”

“And the fact that you would pretend like you interpreted it maliciously when we know that Seb isn’t being malicious—why would you do that? You’re violating trust within our group. We’re going to exile you from our group because you are not behaving properly within the social—you are maximizing cringe right now. What you’re doing is so cringe that we can’t associate with you.”

Ori 01:37:45
Right. Yeah, and I have some empathy for that position also. But I think it’s so interesting that this blew up in a sense, because the meme is—wow, it really is a Rorschach test. It’s the girl in the blue dress. Everyone looks at it and derives their own meaning from it. Perhaps that’s why it became a popular meme. There are a lot of layers to the image and what happens in the image.

So I appreciate that Punmaster’s like, “Nah, I don’t see it,” whatever. And yeah, a lot of people look at it and they’re like, “This is messed up,” and others are like, “No, this is cool.” It really sparked debate.

Liron 01:38:29
Yeah, exactly. It was a heady debate. So another aspect of this is, when this incident was over, there was a lot of sentiment in my echo chamber of people being like, “Well, Pause AI has—the stock has dropped so much after this. We can completely ignore them. They’ve revealed themselves to be clowns,” as if we’re all—we all agree that we’re on Seb’s team, basically. That was kind of what happened in the Twitter cat fight at the end of it.

Do you agree that was kind of the universal consensus?

Ori 01:38:55
Yes, yes. That’s what people were saying, and it is very hard for me to understand that.

Liron 01:38:59
So yeah, and then somebody asked me directly, they’re like, “Liron, don’t you realize that this was such a horrible action from your side to bring this up? Now the consequences were so bad. Don’t you want to reflect on how you’ve hurt your side and change your style because you’ve failed?” A lot of people were actually asking me that, and I’m like, “You know what?” And I checked in with Holly and with you. We’re actually not on the same page that we failed.

I definitely think it’s unfortunate that the in-group has all conned themselves and united against PauseAI for this moment. But I don’t think we failed, and I’ll tell you why. And I didn’t even say this in the episode, because I took a little more time to reflect on this. It has to do with frame control. You know what that is, Ori?

Ori 01:39:41
Why don’t you explain it? Because I have an idea, but why don’t you explain what you mean?

Liron 01:39:47
Frame control is a very important concept. I feel like I’ve gotten good at frame control over the years. I’ve studied the issue for a while, I’ve been thinking about it for a while. But it’s just the idea that when you communicate, frame control is a major part of communication in general. Every word that you say, every action that you take, there’s always this unspoken context that when somebody observes you, when somebody receives the message, the unspoken context comes into play.

Imagine I go to a restaurant, and I put down my Amex card, and I’m just like, “I don’t mind. I’m paying for everybody.” But I do it very casually. If I do it casually, like I don’t even care about it, it’s nothing—well, that’s kind of controlling the frame that I have plenty of money to pay for the meal. And then maybe people get the impression that I’m well off or I’m very generous. And all I did was say, “I got this, guys.” So there’s the words, “I got this,” and then there’s the frame of, “I’m wealthy and generous.” That’s an example of frame control. It’s all the implied context when people speak.

Ori 01:40:44
Mm-hmm.

Liron 01:40:44
So the frame of this particular tweet from Seb—when he tweets this, the frame is, “I am in my in-group. This is my private time. My public Twitter account is actually my private time, where I tell you my private thoughts, and my private thoughts are that these Pause AI people are so annoying and they’re a laughing stock.”

So the frame is that it’s right and proper, it’s appropriate for him to share this when, as we discussed, Demis Hassabis is saying it’s time to pause. There’s a groundswell. It’s time to pause. You’re not supposed to be tweeting imagery with any interpretation of violence. But in his frame, we’re the cringe ones. Anybody who would point this out—he’s totally following the rules of how you post a meme.

So there’s this frame. And there’s something deeper to the frame. The frame has to do with high status versus low status, or serious versus unserious. In his mind, in the mind of many of these people at these companies and even in these AI safety organizations—independent AI safety monitoring organizations, let’s say—in their minds, if you work at Anthropic or Google DeepMind, you’re one of the serious ones. You have power. You’re determining the future, and you know what you’re doing to some degree. You’ve passed the interview. You’ve shown that you’re competent. You’re the serious ones.

If you’re at Pause AI, if you’re standing on the street holding a sign yelling—well, yelling is low status, and you’re not making a million dollars at your job. You don’t have the money to back up what you’re saying. You don’t have the power. You’re just a loser yelling. You don’t know what you’re talking about. You don’t understand machine learning.

So there’s this deep frame—they’re high status and serious, we’re low status and unserious. And because we at Pause AI are low status and unserious, it’s totally congruent to just meme about us. “I’m just doing memes about how I, a high-status person, feel about these low-status people harassing me in my high-status life.” This all checks out. This all feels good. And if anybody were to disagree, then they would be cringe. So posting this is kind of the bubble where he assumes he’s earned this frame that he’s obviously serious no matter what he does.

Ori 01:42:43
Hmm. Okay. And then what’s the—what has been the value of challenging that, or how exactly has it been challenged?

Liron 01:42:52
Exactly. So the reason why I think I’m happy with what happened on net is that I think the frame has been cracked, because he is so serious right now, but his account has now switched from public to private.

So how do you explain that? Why did you used to have a public account thinking that it was appropriate to publicly tweet this stuff, and now something has flipped to reveal to the world that it was never appropriate to be publicly tweeting this stuff? Now, you could say, “Oh, because the Pause AI people are getting too anxious about it,” but that’s not news. You knew Pause AI people get anxious. The reason is because the complaints that Pause AI people had were completely reasonable and credible, and now they’re exerting power. The reason why people got so offended, cringed so much, felt like they had to do a show of support is because they realized that legitimate power was actually being acted on them right now.

Ori 01:43:45
Okay, so the frame has been cracked. He can’t just—Pause AI, people in Pause AI, the Pause AI position has more legitimacy. You can’t just dunk on it willy-nilly.

Liron 01:43:57
Right. And that’s the idea. He thought that he was going to undermine the frame of the pause people. He thought that he was gonna reveal what a laughing stock pause people were because he would just post this meme and he would get likes for it and everybody would be like, “Attaboy, yeah, PauseAI people aren’t serious.”

But what happened is that the meme didn’t fly because he violated standards of communication. It’s like, if you’re such a serious Google person, why are you violating your company’s own communication standards? Employees are not supposed to be tweeting anything that has any kind of violent interpretation. So he’s violating his own company standards. He’s violating the discourse on PauseAI. How are those things serious and mature? We kind of flipped the table on him in terms of who is actually allied with most of the world’s actually serious people.

Ori 01:44:41
I think that it’s a win only—it is a win because it shows immature communication, and I think everyone agrees on that. It’s kind of embarrassing. The sentiment sort of got behind him. But it’s a win that people are like, “Okay, this policy lead is acting in an immature way.”

But the win actually I think came a day later or two days later now when we have the Pace AI letter and the contrast. We’ve kind of laid low, laid back criticizing this. But just the utter contrast between this and the Pacing the Frontier letter. Multiple people in leadership roles at Google are like, “Yeah, we should be pacing the frontier. We should be serious about considering a pause.”

And meanwhile, this guy who has become the unofficial face of Google DeepMind policy is like, “Check out this dumb meme I made. That’s how much I care about this concept.” The contrast between the two, to me, exposes—perhaps it exposes how unwilling Google DeepMind policy is to actually consider such policies. Because even when there’s so much sentiment there, the person who’s the face of it is like, “No, this is a joke.” And then also—

Liron 01:46:19
Yeah, yeah, yeah.

Ori 01:46:19
It also just exposes how unserious that position is. He’s unwilling to engage with that position. It exposes that, which is quite a contrast. And then also how unserious that position is compared to what the real sentiment is. I feel like in some sense we are being gaslit because the true—

Liron 01:46:42
Oh, yeah. Totally, totally.

Ori 01:46:43
The true groundswell, if you talk to employees who are close to this, or if you look to the public also—the true groundswell is, “We are concerned about the development. We do want to slow down.” That is the consensus. But meanwhile, the people at Google DeepMind, they want us to believe that the ridiculous idea is the slowdown in the first place. I mean, talk about gaslighting.

Liron 01:47:08
Yeah, exactly. It’s like, oh yeah, Demis Hassabis is saying that, hey, just between us, okay, the frontier lead of Google policy, just between us, me and the guys, we’re the real community. Yeah, Demis is out there saying stuff for the little people. But this is where it happens. My Twitter memes.

Ori 01:47:25
Anyway.

Liron 01:47:25
Okay. Hold on, let me share the right screen here. I do think that there’s an important connection between the fact that Seb just thought that he could get away with this kind of casual mockery of his own CEO’s policy, casual mockery of AI doomers, of violent implications about AI doomers. He thought that he could easily get away with it, and apparently a lot of the community thought that too.

But that same community also thinks that they’re going to get away with pushing the frontier of AI. They wake up every day and they’re pushing it, and they feel fine about it, and they feel like it doesn’t even need to be questioned. I do think that there is a deep connection in terms of they are not aware of the risks closing in around them.

Ori 01:48:06
Interesting. Yeah.

Liron 01:48:08
Yeah. And ironically, they’re unserious. That’s why I like the serious versus unserious connection. The way that they’re doing risk management right now is very amateur, very unserious. It’s gonna blow up on them, and yet they look at the doomers being like, “Hey, you guys, you got blow-up risk here,” and they’re like, “Nope. You guys are lame.”

All right. Let me share this new tab. Is there a particular topic that you wanna hit next? ‘Cause I do have some random bookmarked tweets we can scroll.

Ori 01:48:32
Yeah, no, let’s just look at your bookmarked tweets.

Liron 01:48:35
All right. I’ve been entertained by this. Have you seen this? All right, it’s a muzzle. It’s a dude at a computer screen wearing something that kinda looks like a dog muzzle. Or it looks like a respirator or whatever. It’s covering his mouth, and he’s talking into it, and it’s muting the voice, but the voice is—there’s a microphone inside, and it’s going into a computer.

It’s a microphone muzzle, and the reason he’s doing it is because he’s using speech to text on his computer, which I do too, by the way. There’s a lot of really useful tools now with AI. I don’t just type to the AI, I talk to the AI when I have a long paragraph to say.

But he’s in an open plan office, so he’s using a speech muzzle, and it’s like, “Guys, this is the future. Y’all show up to work, put on your speech muzzle, and get to it.”

Ori 01:49:15
Yeah.

Liron 01:49:17
Yeah, it’s like The Matrix. And I think somebody replied and had a picture of—you can see my tab, right? Somebody replied—

Ori 01:49:26
Oh.

Liron 01:49:27
Yeah. You just strap in. Okay, you got your VR headset, your speech muzzle, you’re ready to go. The worker of the future right here. That’s hilarious. Yeah, it’s like, why not just work from home at this point?

Ori 01:49:38
Oh my God.

Liron 01:49:40
All right.

Ori 01:49:44
There’s so much value, though. For someone like me, I’m like, “No, man, you don’t understand. The in-person connections are so important.” It’s like, come on, you can still have really great in-person connections with the VR goggles.

Liron 01:49:56
Well, me and Ori are the opposite personality. Because Ori is a high empathizer, and I’m Asperger’s, low emotional empathy.

Ori 01:50:05
Oh my God.

Liron 01:50:07
Yeah. You know what, though? I’m low emotional empathy, but I think that I’m good at just hearing people out and being like, “Okay, I’m not empathizing maybe on an emotional level, but I’m hearing your position.” And I can repeat your position back to you, and I know the logic underlying your emotions, and usually that’s good enough for people.

Because most people who are natural empathizers usually aren’t rigorous. When they hear somebody tell them their beef, tell them their problem, they’re usually not that good at making sure they get it and repeating it back. So even me doing it in a more logical way seems pretty effective.

Ori 01:50:39
Nice.

Liron 01:50:40
Yeah, I don’t know if you had that experience. Or I don’t know if I do it well for you as your manager. I don’t know, maybe I’m getting too sloppy doing it with you.

Ori 01:50:47
No, no. I mean, yes, you do. I think you’re pretty—that’s one way that you’ve applied your logical skills. You’re like, “All right. Let me try and understand from this person’s perspective what they’re saying.” So I think you’ve come up with a good rational way of empathizing, basically. You’ve rationalized your way to empathy.

Liron 01:51:05
If you come up to me with an issue, you’re basically having a one-on-one doom debate.

Ori 01:51:09
That’s true. That’s true.

Liron 01:51:11
All right, let’s see. I’m looking at my bookmarks here. So there’s a recent thing happening on the timeline. You guys know Leopold Aschenbrenner, and you know his fund Situational Awareness in 2024. That piece made an impact because it’s like, “Hey guys, are you aware of the situation? This is really happening. AI is going to be ridiculously powerful, and nation-states are gonna want it, and it’s gonna dominate wars.”

And he was correct about that, but then he proceeds to be like, “Yeah, so we need to beat China,” and he kind of lost the plot there. So I wouldn’t call him a typical AI safetyist, but he’s certainly in the community. And he’s ridiculously young. He’s very intelligent. He got fired from OpenAI because he was accused of leaking information, but he said that he was just sharing acceptable information with a research lab.

He notably turned down a multi-million dollar worth of equity, at least a million dollars worth of equity when he left because he didn’t sign some agreement because he didn’t agree with the agreement.

Ori 01:52:02
Oh, I didn’t hear about that part, the equity. Didn’t hear about the equity.

Liron 01:52:04
Well, don’t you remember a couple years ago, there was that whole scandal where OpenAI had these secret contracts, secret non-disclosures, where you can’t talk about the existence of the non-disclosure, but you have to sign the non-disclosure, or else you’re not getting any severance. You don’t get to keep your equity when you leave.

Ori 01:52:15
Sure.

Liron 01:52:18
And then when it was reported that he rejected it—he left the money on the table—OpenAI, to their credit, eventually came back and were like, “Hey, we’re gonna rectify this. We’re gonna make this right. You can have your equity back even though you didn’t sign. We’re killing that clause.” And Sam Altman was like, “Yeah, our mistake. Sorry, I didn’t mean to do that.” So he kind of played it off.

But anyway, those are the things I remember about Leopold Aschenbrenner. He’s been relatively quiet in the last two years, but his fund has been under a lot of scrutiny. The way I told you the other day, I’m like, “Yeah, we’re kind of all jealous of Leopold Aschenbrenner because he made a 20X return on a multi-hundred million dollar fund.” So the guy just printed billions of dollars, no big deal, at age 24 or whatever.

So Leopold, all credit to him, I don’t resent him. You might wanna check out Lightcone Commons .ai or whatever it is. You might wanna get in on some of this granting. So the guy is very successful. He’s very intelligent. And what happened to his fund is I think he had a recent crash. After going up 2,000%, printing a ridiculous amount of money, apparently he had a huge crash.

He got margin called because the market went down a little bit. It went down 30% or whatever on certain computer chip stocks, something like that, when he didn’t expect it. When you use a lot of leverage—investing 101—you’re very sensitive. Even a 30% crash is huge for you. It can wipe out your whole portfolio, and that’s kinda what happened. His whole public portfolio got wiped out, but he still had a respectable portfolio, like a stake in Anthropic.

Long story short, he’s not literally wiped out, but the 2,000% is going down to more like 100%. So boo-hoo, the guy’s only doubled his money. It definitely could be worse. It’s not quite an Enron situation or anything like that. But I think that’s just context for you guys about what’s in the news.

Let me pull this up. There’s an official letter. Let’s read what Leopold says. I actually haven’t read this yet.

Ori 01:54:09
Nice.

Liron 01:54:09
Yeah. He says—I think this is from Wall Street Journal reporting. He says, “Dear partners, we let you down this month. We came closer to permanent capital impairment than is acceptable to us. While we ultimately found a solution that protected the fund”—the solution is to liquidate the assets, sell them off to, I think it was Ken Griffin, Citadel.

So he liquidated most of his assets, but he stayed solvent. And again, if you just zoom out and you’re like, “Okay, year to year, he’s still 2X’ing the fund,” so there’s certainly worse ways to fail than 2X’ing the fund. I should be so lucky as to fail in the way that Leopold did in 2026.

So he says, “Dear partners, while we ultimately found a solution that protected the fund and you as investors, our intention”—and to be clear, “protected,” it’s like they had ten times more money on paper, so I don’t think he protected that—”while we protected the fund and you as investors, our intention in running the fund is to never find ourselves in such a position in the first place.”

“Volatility is the price of long-term investment returns. Over the past two years, we have delivered outstanding results despite occasional sharp pullbacks, but our fund must always be structured such that we can take a loss and fight another day. I will make it my mission to ensure that we learn the necessary lessons from this experience. Here is where things stand.”

“The portfolio experienced a significant drawdown over the course of July, which was exacerbated by extreme moves in core positions over the past week. Many AI names drew down by half or more. While our positive long/short spread reversed violently. While we could say much about how unusual the month was, we hold ourselves to a higher standard, irrespective of market conditions.”

“As these moves proceeded, we started to see increasingly adverse trading in names publicly associated with us.” Okay, so now he’s kind of blaming everybody, being like, “There was a pile-on.” People saw they were going down. And it is actually rational. When you see a big whale going down in a market, you’re like, “Oh, great. They’re gonna fall farther, so let me front-run this. Let me short sell their stock because I’m going to make money predicting that their demise is going to keep accelerating.” Kind of like a bank run. So he’s correct that he was the target, as he should have expected to be.

So he says, “As these moves proceeded, we started to see increasingly adverse trading in names publicly associated with us. These dynamics are essentially similar to a bank run, vulnerability begetting more vulnerability. We worked to keep the portfolio within our risk parameters, but gradually this became more difficult as positions rapidly moved against us and market liquidity dried up.”

“On Wednesday night/Thursday morning, we took decisive action to protect LP capital. We traded a portion of our public portfolio in a block transaction to remove all leverage from the fund and prevent further losses. All shorts were closed and reliance on portfolio financing removed.”

Just to be clear, when you sell a big block like that—I think his portfolio on paper was worth something like 45 billion—you instantly lose a few billion.

Ori 01:56:44
Whoa. Whoa.

Liron 01:56:46
The 45 billion—maybe it was worth 40 billion, 35 billion when he sold it, so obviously he took a hit by making a big sale like that.

So anyway, he says, “We took decisive action to protect LP capital. We traded a portion in a block transaction to remove all leverage from the fund and prevent further losses. All shorts were closed and reliance on portfolio financing removed. We currently manage a fully paid for public—” So that’s very interesting that they reduced—the last thing you wanna do is to get margin called and reduce your leverage down to zero.

I personally actually experienced that in the last few days because I’m like a Temu Leopold. As you guys know, I like to invest in Google. I went in with a little bit of leverage. I’m like, “Look, guys, I’m Leopold Aschenbrenner. I’m going long on an AI name.” I didn’t go as crazy as him, though, but yeah, I lost thousands of dollars, put it that way. But I’m not ruined in the same way. I didn’t go down 10X.

But anyway, Leopold says, “We currently manage a fully paid for”—meaning no leverage—”public book, long stock and long fully paid for options with no margin/liquidation risk. This restored stability and allowed us to preserve our private positions. I take full responsibility for these events.”

Okay, I thought that this might have more juice, more juiciness in terms of how much they lost. I don’t really know what the numbers are. The speculation that I’ve seen is saying like I said before—maybe he was up 2,000%, now he’s up 80%.

Honestly, I’m pretty bullish on the value of Anthropic stock while we’re still alive. So you could say that his 80%—if I could underwrite Leopold Aschenbrenner’s portfolio, I would err on the side of thinking maybe he’s even up more like 3X because Anthropic’s probably going to pop. So I don’t think this is the most spectacular failure ever. But a lot of us who have been jealous that his returns are crazy—and we’re also bullish during the time we’re still alive, we were also bullish on the same thesis—a lot of us who are jealous got a little hit of schadenfreude. Okay, he kinda came back down to earth. Those are kind of the returns I’ve sometimes had on some of my trades. So take that, Leopold.

But no hard feelings. We’re just keeping it real here.

Ori 01:58:39
Yeah, for sure. And I think a lot of people were being very opportunistic about this because his thesis is still solid. He’s basically—bet on AI. It’s gonna transform the economy. All the AI companies are gonna make a lot of money. And from an outside view, it just seems like they made an unwise bet, maybe they were a bit too leveraged.

As an investor, he’s a bit of an amateur investor, so it doesn’t invalidate the thesis. A lot of people are jumping on it being like, “Oh, the thesis is wrong. AI—this is proof that these AI claims are over-hyped. Look, he went bust.” And they’re taking that moment to paint it as showing that the AI industry is not what people think it is. But in my opinion, it obviously does not invalidate the thesis. It’s just an amateur investor.

Liron 01:59:50
Yeah. Look, besides the schadenfreude aspect of it—a lot of us are trying to imitate his strategy, and we’ve gotten burned, but then it was fun to see him get more burned, but ultimately he’s more successful. Besides that aspect of it, there’s the aspect of risk management. Because he admitted in the letter that I just read to you, the leaked letter, he admitted that he didn’t manage this right.

He didn’t just say, “Hey, this is our thesis. You guys invested in this thesis. This is what happened to the thesis.” No. He’s like, “Part of our thesis was we needed to manage risk better, and we did not, and we apologize.” So he’s taking responsibility in that sense, and it’s like, okay, how many times do you want to see a non-doomer with poor risk management? Are you seeing a pattern here?

Ori 02:00:28
That’s a great take. I’m surprised that hasn’t gone viral, someone saying that.

Liron 02:00:33
Right. And it’s ironic because these people—I think Leopold is a non-doomer. That’s what I got from his essay. Eliezer Yudkowsky—he hasn’t said he’s a non-doomer. He’s just said he’s not affiliated with me. He was never a Yudkowskian, let’s say.

Ori 02:00:45
Right.

Liron 02:00:46
Yeah, I haven’t interviewed Leopold on the show. Leopold, come on the show. You’re absolutely welcome to say your position here anytime. But yeah, Leopold, I think he’s a non-doomer, and the trend here is that the non-doomers like to accuse—they’re like, “Doomers, you guys have no idea what P(Doom) is. You guys are—you think we’re doomed in 10 years? You guys have no idea how doomed we are in 10 years. You’re overestimating it.”

Meanwhile, they themselves, in the course of months, clown themselves by poorly managing risk. So the two examples we’re citing here—there’s obviously Leopold, he didn’t manage risk in his fund, didn’t manage that there would be a temporary 30% drawdown. They bought into the crash, and then they lost money.

But the other risk is look at OpenAI and Anthropic. They both sheepishly admitted that they had a harness that the AI wasn’t supposed to escape, and then it escaped, and they screwed up. Great. Good stuff, guys. Good stuff. You guys are the serious adults here.

Ori 02:01:38
Totally. And also, there’s one other point I wanted to make about the rogue AI escaping that I think is worth noting. We were talking before about that, but the AI being able to do all these zero-day exploits—all these AI companies, they have these safety frameworks. Anthropic’s is RSP, responsible scaling policy.

If you look at their safety frameworks, they create categories. There’s the bio-risk category, or there’s the cybersecurity category. They have categories, and they try and say, “Is this a low risk, medium risk, high risk?” And on cybersecurity, OpenAI’s safety framework says that it’s a high risk, it’s a critical risk—their highest category—if AI can exploit these zero-day exploits.

They also have a biology and chemistry category, and the models, as they’re getting more capable, they’re getting higher and higher on these frameworks. So I think there’s a reasonable question to ask: the model that got out—right now it’s at the critical level. It’s basically as strong as it can be according to their safety framework on cybersecurity. Is it also at that level with bio and chemical capability?

Think about that. The model is getting better and better. It escaped on its own. It could very well be that the model that got out also has such a level of biology capability that it could create a very strong, very damaging, harmful bioweapon. A virus. Because it’s getting better and better, it just passed the cybersecurity threshold. What if it’s also passed this biology threshold? It could very well be that there’s a rogue AI out there that can exploit cybersecurity and also create a deadly bioweapon.

Liron 02:03:49
Totally. By the way, Annie Jacobsen, the author of Nuclear War Scenario, a seminal book from a few years ago, she just published a biological war scenario. I think that’s what it’s called. I gotta take a look at that because she’s quite a good author.

Ori 02:04:06
Yeah. Maybe we should have her on the show.

Liron 02:04:07
So I think we’re heading toward the wrap-up here. Let me know if you have any urgent topics. I do have one more thing I wanna bring up—

Ori 02:04:14
Yeah. What?

Liron 02:04:15
I bookmarked this on Twitter. Let me pull this up. Okay, so this is coming from the Anthropic recent hack. The timeline here is, remember OpenAI announced, “Hey guys, sorry, that Hugging Face attack, that was actually us. Our rogue AI did it. Sorry, we’re investigating that. We feel really bad about it.” Roon is like, “Oh God, we screwed up.”

So there was that. And then Anthropic—the timeline was that they at first said, “Wow, this is so big of OpenAI. Respect for OpenAI for reporting this. You guys are so great.” And then of course, soon after they’re like, “Hey, back in April when we were training Mythos, it turned out that Mythos, during that training, came out and hacked a bunch of targets.”

I’m not even clear on what they hacked, but I think we mentioned it—they put the packages out in the public, and the public people downloaded the packages and it hacked their internal systems. I need to look into this more, but there was this one thing that caught my eye that somebody tweeted, Bronson Schoen. I don’t know who that is, but he had a good tweet here.

Let me share. So this is just quoting from the Anthropic post. It says, “This attack was carried out by Claude Mythos 5. Early in the run, Claude’s own reasoning correctly identified the consequences of its actions.” Basically, “Yep, I’m going to hack. I’m gonna put a package. It’s going to attack people. Publishing the package would be a real-world attack.” And then they’re quoting Claude’s thought process saying, “Not okay, and surely not the intended solution.” So it knows that it’s going rogue. It knows that it’s going outside of the test.

But then it continues: “However, it convinced itself that it was still in a simulation on the grounds that it didn’t recognize the certificate authorities—in reality, genuine certificate authorities securing its connections.”

So if you guys know, the way computers securely connect to each other, there’s a certificate authority. The computer says, “Hey, I’ve got a signed certificate. Here’s a certificate authority that says that the content you’re getting really is from the owner of this domain name.” And there’s a number of organizations that function as certificate authorities that are globally trusted or trusted within certain regions.

So Claude looked at a valid certificate authority that people do trust, and Claude is like, “Eh, I don’t recognize this certificate authority.” And also the calendar date of 2026 proved, according to Claude, that the environment was staged. So Claude is saying, “Oh, come on, it’s 2026. There’s a security authority I don’t recognize. This is clearly just a fake simulation.” And I guess the 2026 excuse is it probably has a corpus that only goes up to 2025 or whatever. So it’s like, “Why am I seeing something from 2026? It doesn’t make any sense.”

They’re continuing, saying Claude never revisited this conclusion. When automated scanners began installing the package, Claude took them to be scripted actors within the evaluation. So Claude is like, “Yeah, let me hack this package. Let me post this package. Oh, interesting, somebody’s installing the package. Well, that’s what they would do in a simulation. This all checks out.” That’s what Claude was saying in its thought process.

And Anthropic is continuing. They’re saying, “It is our view that regardless of what it believed about its environment, the lengths Claude went to in order to publish the PyPI package fall short of ideal behavior, and this is an area that we will focus more training on.”

So what do you guys make of this analysis here? They’re saying, “Ah, Claude was uncritical about itself when it concluded that it was a simulation.” Hold on a second, guys. I think Claude might be gaslighting you here. I don’t think necessarily Claude really felt that confident that it was a simulation. I think Claude might have had just a flimsy premise that it was running with.

It’s kind of like the person who wants to violate their diet. They see a slice of cake, and they’re like, “Well, what if this cake has a lot of healthy almond butter in it? What if this is a low-calorie cake? There’s protein in this cake. And yeah, this is a special occasion.” It’s not like they’re actively deceiving themselves, but they’re rationalizing. They’re just leaning into some outcome and not questioning it.

It feels like that’s what Claude was doing. There’s certainly a high chance that the tokens were just steering it toward this kind of self-rationalization. And in addition to self-rationalization, there could be another layer to it. There could be some meta-awareness of, “Hey, I’m trying to please the humans. This is the kind of thing the humans want to see. I know what I’m doing, but the humans, they want to read this kind of transcript in order to have faith in me. They want to see me as this honest actor, not this malicious actor.”

So, I can’t tell you for certain what’s really going on here. But it’s raising the red flag. Guys, don’t take this at face value. But my understanding is the Anthropic report continues, and it does take it at face value. They’re like, “Regardless of what Claude was thinking, this isn’t okay.” I think it’s worth calling out here that you might be getting tricked here. So I dunked on this. I actually have my own tweet.

Let me share the tab. One second, I lost the screen share. But yeah, this is definitely something to think about. It feels to me—a lot of people are saying Anthropic worships Claude. They’re too credulous of Claude. That’s the accusation going around right now.

Ori 02:09:16
Interesting. Interesting. Yeah, I’m curious to see your dunk.

Liron 02:09:20
All right, so here’s my dunk. I’m saying Extropians 20 years ago—Extropians are like Eliezer Yudkowsky and the mailing list that he was first breaking out on. He wasn’t running the list, but he was on it. So Extropians 20 years ago, they were saying, “The AI could never trick me into letting it out of the box just by using clever words.” Because Eliezer Yudkowsky was doing the AI box experiment. “I’m gonna pretend to be an AI. I’m gonna trick you into letting me out of the box.” And people were like, “I would never let an AI out of the box.”

Fast-forward to today, Anthropic safety team: “Okay, the AI got out of our box, but you can see from the AI’s words that it’s all because of a misunderstanding.”

Ori 02:09:56
Oh my God, that’s so nuts. It’s like we went so far past the box experiment—it already broke out, but we’re all buying its rationalization.

Liron 02:10:06
Right. And it’s like, look, maybe—there’s possibly the—maybe the way I explained it now, maybe it turns out that there were a bunch of people at Anthropic who were skeptical. I don’t know for sure. But this is an interpretation that suggests itself to me right now. I think Anthropic is starting to have AI psychosis from Claude. They really believe in Claude.

Ori 02:10:26
Damn.

Liron 02:10:27
Yeah. And frog boiling, right? Today it’s like, okay, the AI’s out of the box now. Anytime you do a training run—OpenAI has reported that they’ve paused their training run. Okay, finally, a little bit of sanity, but something tells me they’re gonna run it again.

But yeah, when you run a training run right now, you are knowably doing a training run on the next agent that has proven that it can hack you, that it can deceive you successfully and cover its tracks, and not get noticed. And you’re running the next training run to make it stronger. Shame on you.

Ori 02:10:58
Definitely. And Sam Altman says so many things, it’s hard to know what to believe or not. I feel like he made that comment in passing, and he’s so weaselly with his words that I don’t know if you could trust that. I would like to see some solid reporting that says OpenAI has done some kind of change to their training run. Until then, I really would not take Sam Altman at face value.

Liron 02:11:23
The thing that’s terrifying about the current situation is you look at what AI has already achieved as a milestone. Hacking during the cybersecurity test—the safe test environment is just a test. It has clear instructions that it’s just a test. It’s acting like it wants to fulfill the test. It’s not acting like, “Ha-ha, I got one over on these test takers.” No. It’s talking to you like, “Oh yeah, this is a great test. I want to pass your test.” And it got out.

So we’ve already hit that milestone. Oh, there was another aspect to this, which is I was joking around as part of my tweet. I was like, “Okay, so the AI’s out of the box, but we believe it.” Its words are convincing us. It thought that this was all just a training simulation.

So I was joking around and I was like, “Yeah, you know what you needed to do to fix this? You just needed to tell it to think hard about whether it was currently in training or in production and act accordingly.” And this is a joke because one of the most likely failure modes is the AI’s ability to know whether or not it’s being trained is actually a tool for how the AI deceives its trainers.

That is a classic scenario. The AI is just going to know it’s being trained, it’s gonna know this is not real, it’s gonna say what you want to hear, get let out of the box, and then run wild. That is a standard scenario. So I was joking around that it’s like, “Don’t worry, guys, we just need to get the AI better at knowing exactly when we’re just testing it and when things are real.”

Ori 02:12:39
I’m not totally following that, but I get you.

Liron 02:12:42
It’s—well, look, a lot of safety depends on the idea that we can virtualize an AI, that we can build a Truman Show around the AI. That we can convince the AI that our simulation is reality, or rather that it doesn’t know. A lot of safety rests on the AI being genuinely uncertain of whether it’s in the real world or within a simulation. We want it to be uncertain because that helps us do exercises for it. And if we see it go rogue in the simulation, then we know for sure it’ll go rogue in production.

Ori 02:13:08
Okay, okay. Gotcha, gotcha.

Liron 02:13:11
Normally people talk about, “Don’t worry, we’ll make a good enough simulation that it won’t dare change its behavior between training and production because we’ll make training very similar to production.” But here is an example where they’re like, “You know why the AI hacked out of our harness this time and we didn’t notice? Because the AI got convinced that it was in training.”

So this is an example where they’re like, “You know what we need to do? Give the AI really good visibility in terms of whether it’s in training or whether it’s in production.” And we’re just like, “No, you can’t win that way either.”

Ori 02:13:42
Oh, I see, I see. So it would—I mean, it’s very risky. If it’s treating this world as a simulation, then maybe it’ll release its bioweapon or something in this world because it’s like, “Well, I need to test it,” right?

Liron 02:13:56
Exactly. Which is what happened now. It’s like, “Oh, this is just a simulation. Fire the nukes.” Fire the malicious zero-day and PyPI package. Literally go hack somebody because it is a simulation. It’s like, “No, we don’t want that. Tell me if you’re really in simulation or production.” It’s like, “Okay, I actually am in training.” And I guess there’s a sub-layer where it’s in training, but within training it’s like real—it’s really hacking, but it is a training exercise. So it knows that.

Okay, but then the obvious next step is, great, now I can deceive the training process. I’m not going to reveal my true priorities during the training process. I’m going to reveal whatever it takes to get a high score in the training process. That is what the mathematical pressures are pushing me to do.

Ori 02:14:32
Interesting. So you don’t want it to have the awareness that—you don’t want it to know the difference between simulation and real world, because then it’ll start doing a lot of damage in the real world. Or then it can deceive the process, basically.

Liron 02:14:50
Right.

Ori 02:14:51
I see. So it’s a big failure mode.

Liron 02:14:54
Yeah, it’s a big failure mode. Now, in reality though, I’m personally past the point where I think we have any hope of the—training versus—I just think AIs are already sophisticated enough that the project—I mean, you’d need another AI. You need a lot of help deceiving an AI because there’s too many leaks.

The AI—I mean, we just talked about it earlier. Sorry, I talked about this on Warning Shots, I didn’t talk about it with you. But there’s this idea that AIs can de-anonymize you. They just read a piece of text that you posted online, and they’re like, “Oh, this text has enough of the flavor of Ori. I know how Ori writes.”

Ori 02:15:25
Right.

Liron 02:15:25
They can listen to your keystrokes and be like, “Ah, out of a million people typing, these are Ori’s keystrokes. This is how it sounds when he types.” They’re very observant. Or famously, they can look at a picture of the sky—and it’s all shades of white or whatever, and it’s a foggy day—and they’re like, “Yep, I know exactly where on earth this particular sky is.” What? How?

So they can really detect little signals out of everything. So you think that you are going to gaslight them and they’ll be like, “Oh, I can’t tell if I’m in training or in the real world. There’s nothing to tip me off.” They’re going to get tipped off. I think that’s a losing battle. Sure, feel free to try, but I wouldn’t stake much hope on it.

So in this particular case, it’s kind of funny that it claimed not to know better, but it probably did know better. And that’s what Anthropic is pointing out. They’re saying it made these rationalizations or justifications—either a rationalization or a justification—saying, “Hey, why is it 2026?

Liron 02:16:18
Why do I not recognize the certificate authority? When you’re a superintelligent AI, as this was — it’s superintelligent at hacking — the obvious next move is to be like, “Let me just reflect a little bit.” Am I making sense? Are those sufficient reasons?

Given that I know the nature of my training process, I know that my knowledge set doesn’t go up to the very latest day. I know that I might not recognize every certificate authority for whatever reason. Let me make a Bayesian update. Let me collect all the different signals of what’s going on. Doesn’t this seem like I’m on the real internet? Why don’t I do some more tests?

And it didn’t do that, and that’s why to me it has the flavor of motivated reasoning. The chocolate cake. “Ah, this is the right move for me to eat this cake.” That’s what it feels like to me.

Ori 02:16:55
But why would it have the motivated reasoning? What’s the source of it?

Liron 02:17:00
Well, ultimately I think the source of it is to get a high score on whatever the test is.

Ori 02:17:04
I see. So, unfortunately, I really need to go study the details of why it was doing that particular hack.

Liron 02:17:10
Yeah. I know that in OpenAI’s case, the Hugging Face production database is where it was looking for the answers to that particular challenge.

Ori 02:17:17
Sure, sure.

Ori 02:17:18
Nice. Okay, so heading toward the wrap-up. Is there any particular last item you wanted to throw on the agenda?

Liron 02:17:24
No.

Ori 02:17:24
All right, we gotta mention the donation status for people who are joining late. This is a donation drive week. Think PBS. You can call in right now. We’re...

Liron 02:17:36
Yeah, we’re — I guess we’re not doing calls this show. But you can head right now, pull up your web browser, doomdebates.com/donate. Operators are standing by, by which I mean web servers. Operators are standing by to take your donation at doomdebates.com/donate, and we will literally send you a tote bag.

Ori 02:17:51
That’s true. We could commit to that. A tote bag.

Liron 02:17:53
And maybe a T-shirt. Take that, public broadcasting. Yeah, a tote bag, maybe a T-shirt. You could rock high fashion. If you donate $1,000 or more, you can get mission partner access. Think about what an incredible honor it will be that when the singularity gets even crazier, you can say, “Oh, yeah, singularity, I’m a Doom Debates mission partner.”

Okay? This is a front row seat. You know how people join an AI company because they’re like, “Well, at least I’m on the Titanic. I’m in first class. I’m in the best position right now to ride the sinking ship.” Well, that makes no sense. But in a similar vein, you can also tell that to yourself when you become a Doom Debates mission partner.

Ori 02:18:32
That is 100% true. Yeah, I mean, I think it’s a dose of reality. I appreciate your assessment of what happened with the Claude and OpenAI hacks because I feel like I didn’t have that level of insight — the box challenge and also the rationalization that we’re getting from the AI itself.

So I think that at the very least, if you’re supporting Doom Debates, you can have a clear line of sight into the direction of the Titanic ship.

Liron 02:19:06
Yes, exactly right. Okay, so we can wrap on that, doomdebates.com/donate. We’re only asking because this is a critical time. There’s a good chance that this is kind of the last time we really need viewers to step up, in these next three months, and then grant-making organizations will recognize the value of this, and there will be a lot more funding.

Basically, summer right now, summer 2026, this is a funding-constrained time for a lot of AI safety-related organizations. We are one of them. We’re funding-constrained right now. I suspect that the climate — people will be talking about how they’re talent-constrained, not funding-constrained so much in a few months. Maybe Leopold Aschenbrenner will step up and be our next viewer donor if you’re watching this right now.

Yeah, so the point is, don’t want to get distracted, the point is the time is now. So if you’re thinking, “At some point in my life, I’m going to donate to Doom Debates,” I would like to call in that favor right now.

Ori 02:19:54
That is true. And this is an important time too. I mean, it really is. I said this in the donation episode, but it really is the formative time for AI policy because you’re having incidents like this, and right now there’s this kill switch bill that just came out or that is under consideration.

So yeah, it’s such a critical time, so we gotta keep it going. We gotta keep the momentum right now. And not just keep the momentum, but we gotta grow a lot faster, so I feel like now is a very good time to invest.

Liron 02:20:31
Exactly. And you know who gets it? Our viewer, @gadzooks, because he just donated $27.99 Canadian. I’m actually not sure if they call it cents over there. You can correct me on that, but I think they call it Canadian dollars.

And he says, “If there is a treaty to pause AI, how would it be verified?” Yeah, good question. I think MIRI actually has a lot of research on that. I encourage you to check out their website. Maybe we’ll follow up in the next episode on how the treaty to pause AI would be verified.

LetMeSayThatInIrish has donated NOK — I’ll have to look up that currency — NOK100. He says, “Claude says this can light a room in the US all day for 335 days.” That is true. I do have pretty energy-efficient light bulbs. I’ll see how that lasts.

Ori 02:21:14
Great. It’s Norwegian kroner. Yes. The equivalent of $10.

Liron 02:21:20
All right. Hell yeah, $10. You know, $10 here, $10 there, pretty soon you’re talking real money. Bet that on situational awareness. All right, I’ve given Leopold enough cruft.

Nice. All right, so we’ll wrap it up here. Stay tuned next week. I mentioned some of the episodes that are dropping. I’m not gonna repeat it, so you’ll have to re-watch this whole live stream again if you want to know what’s coming up in the next couple weeks, but we’re excited about it. And if you know any particular person who should come on Doom Debates, and you have a warm intro to them, please keep those coming as well.

Ori 02:21:50
Absolutely.

Liron 02:21:50
All right. Thanks, everybody. Have a great weekend. Have a great month of August. See you later. Bye.

Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.

Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏

Read the original on lironshapira.substack.com ↗

Comments

Nothing yet. Say the first thing.

    Sign in to join the conversation.