You can find all practical details at Post-AGI.org. San Diego, CA, 3rd Dec. The rest of this post shares some personal reflections on why this workshop matters.
Why even run this? My personal motivations: I’d like a better plan for Phase 2, some maps leading to good outcomes, and broadly more progress on macrostrategy.
Phase 2 problem
As one shorthand, I like Zvi’s framing of what we discuss in the gradual disempowerment paper - even if we mostly solve single-single AI alignment, we face what he calls the ‘Phase 2’ problem:
One term I tried out for this is the ‘Phase 2’ problem.
As in, in ‘Phase 1’ we have to solve alignment, defend against sufficiently catastrophic misuse and prevent all sorts of related failure modes. If we fail at Phase 1, we lose.
If we win at Phase 1, however, we don’t win yet. We proceed to and get to play Phase 2.
In Phase 2, we need to establish an equilibrium where:
AI is more intelligent, capable and competitive than humans, by an increasingly wide margin, in essentially all domains.
Humans retain effective control over the future.
Or, alternatively, we can accept and plan for disempowerment, for a future that humans do not control, and try to engineer a way that this is still a good outcome for humans and for our values. Which isn’t impossible, succession doesn’t automatically have to mean doom, but having it not mean doom seems super hard and not the default outcome in such scenarios. If you lose control in an unintentional way, your chances look especially terrible.
Unfortunately, we have even less of a plan for how to play Phase 2 than for Phase 1.
To develop one, we need a better idea of what civilization we’re aiming for.
Active inference and maps pulling the territory
Active inference suggests that minds like ours minimize prediction error both by adjusting our world models and by adjusting reality to match those models. It’s a bit more complicated than that - if you are technically inclined, you can check our paper on the Path Divergence Objective for an attempt to model what’s going on. One practical upshot: in case of humans, our maps tend to pull territory in their direction.
The AI safety community has extensive experience with this phenomenon. Partially self-fulfilling prophecies in AI strategy include concepts like AGI, the AI race, AI securitization, and the US-China superpower competition. We’re quite good at producing detailed maps of failure modes, and then watching them unfold.
What if, for a change, we focused on finding maps that lead somewhere we actually want to go? It is probably way harder; on the other hand, fewer people are trying.
Macrostrategy is neglected
There are a few places you can work on macrostrategy full-time (btw ACS is hiring, also Forethought), but the field remains small and needs more interdisciplinary expertise. The questions we face about governance, economics, philosophy, and the future of human agency in a post-AGI world – demand technical insights from a range of fields.
A workshop. We ran one on a similar topic in Vancouver, and it went well (Summary and some of the talks). It helped to surface some interesting ideas, many interesting conversations happened, and overall we felt we should continue. So, hope to see you in San Diego (3rd Dec)
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.