I am starting to feel the AGI

It's a little embarrassing to admit this, and maybe my views will change over the next few months to a year, but I have been "feeling the AGI" these past few months and it felt like time to fess up publicly.

The most recent thing to shock me out of my stupor was my friend Leo's mini-book, Situational Awareness. I don't agree with everything there, but the core story about the trajectory of AI progress over the next few years is currently my modal AI progress outcome. My version of that story is:

• Rapid research progress will continue for at least the next 3-5 years.

  • We aren't hitting diminishing returns to scaling.

    • People like Dario Amodei, Ilya Sutskever, Gwern, etc. who've been most right so far about the pace and shape of progress so far are repeatedly saying this. Folks who've gotten peeks at the next generation of models say similar things.

    • There seems to be enough readily available data to support naive scaling through 2027 or even 2028, and that's without tapping into more expensive sources.

  • Alongside continued scaling, 100s-1000s of the most talented researchers are feverishly working to find efficiency boosts and other "unhobblings" to improve the performance of the models. 

    • There are already signs of this investment working. Context windows are exploding. Gemini 1.5 Pro has a 1M token context window, Claude a 200k one, and Google has said they may soon have a 10M token context window model available.

    • Post-training improvements to GPT4 have substantially improved its mathematical and scientific reasoning abilities. There's reason to suspect the same is true for Claude Opus, although we don't know for sure.

    • Agentic scaffolding is starting to work for coding and potentially other tasks, and these methods mostly don't seem to do fine-tuning. See: [Devin](devin.ai), SWE-Agent, Aider, Lindy, etc.

    • There are rumors, although they are just that for now, that some combination of test time compute and "real" RL during language model training is working. There are also promising-looking papers  Less speculatively, there are a bunch of proofs-of-concept  (Quiet-STaR, MCTSr) or methods that use synthetic data or variable compute to improve performance. Most of these won't work at scale, but only one or a few have to.

  • The core scaling recipe seems to transfer to many distinct domains.

    • Robotics: RT-X, PaLM-E, various humanoid demos from Figure, Tesla, Unitree, etc.

    • Multimodality: language, vision, audio, and video are all being combined in systems such as GPT-4o and Gemini. Scaling also seems to work in each of these domains individually. See, for example, SORA.

    • Biology: We are in the early days, but at least for molecular modeling, we see signs that scaling works.

• Because of this, there's a good chance we'll have AGI, defined as a drop-in for a human remote worker, in the next 5ish years. 

• This will be a really big deal, potentially one of the most important things humanity ever does.

  • Once we have a drop-in human remote worker, one of the first tasks that will get dramatically sped up will be AI research. This will further accelerate the rate of AI progress, rapidly pushing us beyond human level for programming and other ML research tasks.

    • We don't know where diminishing returns or a plateau will be hit but there's no reason to expect it to be that close to human assuming this involves going beyond the imitation learning paradigm.

    • While a drop-in remote worker would be a big deal even if it didn’t dramatically speed up AI research, it would be especially important if it did.

    • Some reasons to think automating AI research will be relatively early in this process: AI researchers are a big chunk of AI company costs, making them an attractive target for productivity gains and eventual automation; some early signs this is possible (maybe cite the work where AI models develop parameters for robotics, eg, Eureka); AI companies have lots of internal data on what their researchers are actually doing along with model performance data - in the language of pharma companies, we might say that AI companies have powerful “clinical endpoint” data on models along with tons of data on the whole process leading up to them, which may not be true for equally valuable tasks; 

  • If a drop-in remote worker happens, it seems likely to me that we'll see rapid scientific and technological progress, especially if AI speeds up even more. This could be incredibly exciting but also comes with risks especially during the chaotic period during which the most rapid progress and deployment is happening.

There are many arguments against this, and I may cover them in another post. But here I instead want to focus on some miscellaneous questions I’ve been pondering when I inhabit the headspace where I really buy this story. I’m honestly quite uncertain about these implications right now, so part of the reason I’m writing this is because it may act as a beacon for others grappling with these same questions. (If this is you, don’t hesitate to reach out.)

Miscellaneous musings

Is bio "so over" or "so back"?

I work on bio. In the world where this story holds up, how much will the bio work we do today matter? 

I convince myself it does matter with the following story: Even if we have AGIs automating research, they'll still be constrained in what they can learn from data. Therefore, it's important we build the infrastructure and tools to generate the data they'll need in order to do research/work that otherwise might not happen quickly enough to matter during a chaotic, rapidly evolving situation.

As a concrete example, if we think it's important that AGI cures all diseases and potentially helps us genetically engineer humanity to be smarter, wiser, and kinder, we shouldn't just wait for AGI. We should be aggressively shaping infrastructure and institutions to remove all roadblocks to biological research and deployment of novel solutions. The FDA's recent platform therapeutics regulation proposal is a good example of what this could look like. It ensures that if future AGIs invent an improved delivery or editing modality, we don't have to wait a decade for it to generate 100s or even 1000s of therapeutics. Similarly, if we think that even superhuman AI will need real-world human data, we should be generating that data now so that AGI can learn from it vs. having to wait for years. We should accelerate automation so that if humanoid robots lag, we can still benefit from early superhuman AI. And so on and so forth.

But much of the “current work will still matter” story feels shaky, and I can convince myself about as well of the opposite case: Early AGI is able to much more intelligently take advantage of existing data and literature. This makes it vastly better at target discovery than any human today. Also, because it's superhuman at programming, it advances the automation state of the art rapidly ala Tinker. Together, this allows it to design proteins and interventions that are extremely safe in humans, so it's able to deploy these interventions widely. In this world, AIs can design drugs like humans currently build physical infrastructure - they can design ahead of time and be confident it works. Regulatory barriers aren't a huge issue either once it demonstrates this ability to design near perfectly safe interventions.

As a result, in the same way that drivers aren't a complement for self-driving cars and human computers aren't a complement for digital ones, current drug developers and biology practitioners mostly just become obsolete. 

The point isn't that either of these stories describes what will happen. They both lack sufficient detail to judge. But as broad sketches, it's quite hard for me to rule either out. Hence I'm left unsatisfied.

A meaning meat grinder

Stepping back even further, making sure the hand off to robots who will forever shape the future may provide meaning to some people (perhaps those who get tapped to work on The Project), but for me it mainly serves to drain meaning from my actions today. I derive at least some meaning from feeling like I'm working on things that will matter in the future and more broadly feeling like my and other humans' agency will continue to shape the future. Acknowledging the likelihood of superhuman AI forces me to grapple with the question of how much of my motivation derives from a belief that what I do now will matter in the future.

Now one counterargument to this is that I personally don't really shape the future that much today, and assuming AI goes reasonably well, there's not a massive difference between (sentient) AIs shaping the future and humans. 

I don’t really buy this. At risk of self-aggrandizement, I work on things I hope can contribute to potentially curing genetic diseases as well as complex ones. Even if the company I work at doesn't succeed, I have high confidence that something like it, done by similar-enough people, will be transformative, and I can vaguely feel that I'm part of it.

Zooming out, considerations such as the above let me feel like I am part of the story of humanity shaping the future. I live in the most important, greatest country on Earth, and at least feel kinship with the groups and people working on some of the important problems and technologies of the day. On the other hand, if AGI really happens, the future will mostly happen elsewhere, both literally and figuratively. Literally in the sense that I assume AGI would quickly venture out into the cosmos at speeds and through means incompatible with biological life. Figuratively because AGIs will think much faster than us and leave us behind intellectually so much of what's happening in science, technology, and even (AI) society will likely be totally inscrutable to me (and you). 

This scenario feels more like getting left behind in a zoo than participating in the future. 

What about the uploads?

Both Leo's posts and the above tacitly assume that humans continue on as biological entities. In practice, when I talk to truly AGI-pilled folks, a huge fraction of them seem to assume that a successful AI takeoff would involve solving mind uploading, allowing us to become digital intelligences alongside AGI. 

There exist various permutations of this. Elon and Sam Altman both talk about the merge, where BCIs allow us to become AI's limbic system. Robin Hanson writes about ems being the most likely form of humanity to survive and flourish in the future. Holden Karnofsky writes about “digital people” as a way to unify AIs with personhood and potential mind uploads. Bostrom and Sandberg have written about various forms/options. All the permutations share the key property where humanity radically evolves in a way that potentially enables keeping up with or transforming alongside AGI.

I suspect Leo didn't write about this because it feels even more fantastical than the other, already Overton Window breaking parts of the story. But even if it's out there or weird, it feels important for me to discuss. Without it, humanity's disempowerment feels nearly inevitable (even if AGI goes reasonably well), whereas with it, we have the potential to continue to be part of The Story happening in our lightcone. If you buy those mind uploading scenarios, and also value human agency, perhaps you want to work on accelerating those technologies, or facilitating their acceleration by better AI in the near-future, such that they can better keep up with AGIs.

Conclusion

The more “AGI-pilled” I become, the more uncertain I become about the implications. Thinking about what happens even if AI goes right has been surprisingly dislocating, and I don’t feel like I’ve made much progress yet. 

I suspect and hope that this year’s progress will provide substantially more information about the trajectory we’re on if we get to see the next generation of models (e.g. GPT5), and I expect to continue to think and write about these questions over the next 6 months. If you’re also thinking about them, don’t hesitate to reach out.