Intelligence Explosion Macroeconomics

In this blog’s most read post to date, I presented a basic economic model of AI existential risk. The headline result is that if it’s possible for an AI to kill everybody and take their stuff, then it’s optimal for it to do so. Importantly, I didn’t need to build a model from the ground up to show that result. Taking a standard economic model and tweaking it only in allowing for the possibility of an AI seizing control is sufficient to show that the incentive to do so is present for most utility functions.

A natural objection to this is that even if an AI would like to unilaterally acquire all resources, it won’t be able to. After all, many people would also like to take total control of the world, but so far none have succeeded. In the post, I said I was pretty sure I could model how an AI could accrue a massive technological edge, but it would be a topic for another post. Well now it’s another post.

Fortunately for me, I don’t have to be the one doing that modeling. Phil Trammell and Anton Korinek have written a thorough paper exploring growth dynamics of powerful AI. They go through a variety of models, and show that you don’t need strong assumptions to show rapid growth from AI. Taking basic models and letting AI substitute for labor and/or drive technological progress is largely sufficient.

While the Trammell and Korinek is certainly the most extensive paper on AI-driven growth to date, it builds on existing research from prominent economists, notably Aghion, Jones, and Jones (2017) and Nordhaus (2021).This post will be aimed at reviewing a selection of models from that body of work, to argue that AI is likely to become capable of taking all resources from humans.

Economists often have an aversion to models suggesting rapid increases in growth. If a model of the economy matches historical data but predicts sharp changes in the near future, that can be more of a knock against the model than a reason to anticipate genuine transformation. That concern is often reasonable, for example in cases like the cubic model of COVID forecasting, which lacked a mechanistic explanation for the change. The point I would like to make in this post is not only that economic models predict a sharp takeoff once we have AGI, but also that economists should take that aspect of these models seriously.

Before we get into the models, it’s important to address a few conceptual points.

I’m treating “AI developing a large technological edge” as equivalent to “AI developing world takeover capabilities”, but are those the same thing? For a sufficiently large gap, this is obviously true. A matchup between a modern military today and early hominids would be no contest. However, a differential of that size is hardly necessary. In the Gulf War, the casualty ratio between Iraq’s forces and the opposing coalition was roughly 100 to 1, with only a technology gap of around 10-15 years. Certain technologies, like bioweapons, further disfavor biological parties, and general AI is itself a technology with military uses.

To be clear, I don’t think a rogue superintelligent AI system would literally wage a conventional war on humanity. Instead, that possibility acts as a floor, indicating that it could take over at least that efficiently. I would expect a more subversive plan in practice, one that culminates in a sudden loss of control.

The second conceptual point is that in economic models, predictions of indefinite exponential growth are extremely common. As a group that parrots slogans on the internet likes to bring up, infinite growth isn’t possible in a finite universe. This is not a compelling knock against economic models. While there are technically fundamental limits to growth based on physical constraints, those are incredibly far in the future.

Aghion, Jones, and Jones (2017) defines two kinds of intelligence explosions: Type I occurs when growth is exponential and the rate of growth itself increases over time, while Type II has the rate of growth increase so quickly that it can reach infinite growth in finite time. Neither should be taken as a claim of literally infinite growth, but both imply enormous gains relative to an economy not undergoing such an explosion. Achieving the technological edge required to dominate other economies does not even require such a growth explosion, a simple rapid increase would be sufficient.

The final point to clarify is that this post addresses artificial general intelligence, which has the capability to do all tasks a human can do. Many people, especially economists, are skeptical of rapid growth from AI due to Baumol’s cost disease. If some tasks remain unautomated, those will become the bottleneck, and slow down future growth. For example, calculators drastically increased productivity in rote arithmetic tasks, so now rote calculations take up approximately 0% of the budget for new projects.

While I think this is a reasonable point to make regarding immediate term forecasts, it doesn’t engage with the premise of the discussion. There are many tasks that AI can’t do currently, but the point is to anticipate what happens once that is no longer true.

Relatedly, this general ability extends to embodied tasks too, not just what can be done remotely where current models are limited. Is this realistic, given current robotics technology? It seems like the current bottleneck in robotics is software, which is why good robots are accused of being teleoperated. But also, we’re not interested in the limits of current robotics. Time will pass between now and the creation of AGI, in which the state of robotics will advance. AGI, even if poorly embodied, will be able to do robotics research that can advance it further. If some physical labor is a prerequisite for robotics advancements, that can be done by humans who are bribed, threatened, or manipulated by an AI. While embodiment may present an initial obstacle to AI growth, it should not be seen as a hard constraint.

The Trammell and Korinek paper starts by pointing to the 2021 paper from Nobel-winnning William Nordhaus, Are We Approaching an Economic Singularity?, which uses one of the workhouse models of economics: the constant elasticity of substitution (CES) production function.

Here, Y is the total output, K is the amount of capital, L is the amount of labor, with A and B representing how much capital and labor are respectively augmented by technology. The exponent, ρ, determines whether capital and labor are substitutes or complements

When ρ is less than 0, capital and labor are complements, and a change in their relative costs leads to a less than a proportional change in spending on each. When ρ is greater than zero, they are substitutes, so a change in their relative costs results in a greater than proportional change in spending. For ρ exactly equal to zero, we substitute in a different function where a change in relative costs does not result in changing how much is spent on each.

Nordhaus asks, rather than using a different model to capture the effect of AI, what if it just affects the value of ρ? When ρ goes from negative to positive, capital and labor go from being complements to substitutes.

In either case, capital accumulates at rate sY/K, where s is the percentage of output that gets reinvested, while labor grows at some exogenously given g_L. For now, we’ll assume that technology is constant.

When capital and labor are complements, whichever one grows slower acts as a bottleneck, so the growth rate is given by min(sY/K, g_L), which is bounded by g_L. When they are substitutes, however, the growth rate instead increases at the faster rate, max(sY/K, g_L). If capital is the faster growing input, then it comes to dominate the production function, and Y ~= AK. Then, the growth rate is max(sA, g_L). So, switching the value of ρ from negative to positive has the potential to permanently increase the exponential growth rate.

Now let’s suppose that technology is getting better over time, with A and B growing at rates g_A and g_B respectively. In the complements case, this increases the bound from g_L to (1 + g_L)(1 + g_B) - 1, which is a higher rate of exponential growth. However for substitutes, the rate sA is now itself growing exponentially. As such, both capital and output end up increasing at super-exponential rates. Switching the value of ρ switches from standard exponential growth to a Type 1 growth explosion.

Next up in the Trammell and Korinek paper is a model that incorporates AI in a very straightforward way, based on Hanson (2001). We start with a Cobb-Douglas production function, and allocate capital between standard equipment and robotics that can replace labor.

Here K is capital, H is human labor, S is the share of capital allocated to robotics. We restrict a and b so that they’re both positive and a + b < 1.

Early on, capital is scarce, so it is allocated entirely to equipment, and we have the standard Cobb-Douglas. If technology grows at rate g_A, capital will grow at rate g_A a/(1-a).

Eventually, as the technology improves and capital accumulates, it eventually hits a point where it is optimal to use it to replace labor as well. Then, capital will grow at rate g_A(a+b)/(1 - a - b). Empirical estimates show that this produces a one time jump from a growth rate of 4.3% to roughly 45%. The basic idea at play here can be extended to more complex models, showing a sharp and permanent jump in exponential growth rate once it becomes economically feasible to replace labor with capital.

Next we move on to task based models, where different sectors of the economy or individual tasks can be automated to different degrees. Here, the CES production function is generalized, so that there is a continuum of sectors, indexed between 0 and 1. The fraction with an index less than some β between 0 and 1 are automatable.

If you do all automated tasks with capital and all unautomated tasks with labor (which will occur eventually if capital grows faster than labor), it is optimal to spread each resource equally between all tasks. The production function is then’

Here, A = β (1−ρ)/ρ and B = (1 − β) (1−ρ)/ρ, and this has been reduced to a two-factor CES production function. We could further multiply A and B by a term for technological productivity increases other than automation.

The basic dynamics are exactly the same as the earlier discussion of this model, but this model allows for capturing more advanced dynamics as well. Automating tasks is a new type of growth in this model, and it compounds existing growth sources.

However, the key difference between this and the earlier CES model is that switching from ρ < 0 to ρ > 0 is no longer needed to unlock growth at a rate sA. Here, that can also be done through the automation of all tasks (setting β = 1), so that there is no labor to act as a bottleneck. As in the earlier case, if growth is occurring at rate sA and the technology is also improving over time, this unlocks hyper-exponential growth and a Type I growth explosion.

Finally, we can model the process of technological development endogenously, rather than taking it as exogenously given, and see how replacing human R&D with AI R&D affects growth. A standard model of technological growth takes the earlier CES model, and lets a portion of labor be used to improve technology instead of production.

Y = [(AK)^ρ + (B(1 - S)L)^ρ]^(1/ρ)

g_B = θ B^φ (SL) ^λ

Where S is exogenously given as the share of labor dedicated to research, and θ is a constant. λ > 1 gives increasing returns to research, such as through the combination of ideas, while λ < 1 gives decreasing returns, such as duplicating work. Without labor growth, φ < 1 means that growth eventually goes to 0, while φ > 1 means it goes to infinity. Adding labor growth decreases the φ threshold for growth going to infinity.

To model the effects of becoming able to use AI for research, we can replace the labor only research process with a CES one.

g_B = B^φ [(θ_K K)^ρ + (θ_L SL) ^ρ]^(λ/ρ)

This ends up with similar dynamics to the CES case, although there is insufficient growth in capital to maintain the rate of technological growth when φ < 1 - λ (this would imply sharply diminishing returns to additional resources on growth though). The growth rate is constrained by labor without AI, but if capital and labor are substitutes or if all tasks are automated then the growth rate grows exponentially, which compounds with growing capital for a Type II growth explosion.

It is somewhat unusual and unrealistic to have the research process only increase productivity for tasks done by labor. However, it acts as a floor, and when the productivity of capital also increases growth is even faster.

While this model ignores the effect of AI on actual production, we can combine models of research automation with production labor automation. In those cases a growth explosion from either source causes one overall.

Erdil and Besiroglu (2023) look at a variety of objections to the possibility of a growth explosion, and rebut them both conceptually and empirically. These empirical arguments are made by showing how the parameters needed to not have a growth explosion after AGI are unrealistic. I’d like to address two more arguments that don’t contest the possibility of a growth explosion, but rather question it as part of the story for existential risk from AI.

“Sure, explosive growth is possible, but we’ll be able to intervene early enough to prevent it”

This objection is saying that even if an AGI has the ability to rapidly develop powerful new technology, we’ll be able to catch it early and prevent that. It won’t initially be able to overpower humanity, so if we can detect the start of that process we’ll be in a position to nip it in the bud. The AI control agenda suggests doing so by limiting an AI’s capabilities and monitoring it using less powerful but trusted models. This could be added to with a global effort to prevent rogue AIs from accessing markets, acquiring resources, and building a power base.

A related objection, which I’ll lump in, is that before an AGI is capable of doing all the tasks necessary for explosive growth, it will be capable of doing the task of alignment research. Then, we can have it solve the alignment problem before we create any AIs that are capable of taking over.

These are both active areas of AI safety, so I take this objection seriously. My first concern is that they won’t actually be implemented. If the labs building AGI are unable to resist the temptation to leverage the full capabilities of their models, then it doesn’t matter whether the proposed constraints would have worked. Alternatively, the labs may just not realize that a model they’re deploying is unaligned and powerful enough to pose a threat.

Let’s be charitable though, and suppose we have a guarantee that these plans will be thoughtfully attempted. Is there still cause for concern? I think there’s a chance these plans could work, but only a chance. The impression I get from Redwood Research, who originated the control agenda, is that they favor it because it’s an obvious and tractable step, not because it drastically reduces risk. Controlling a misaligned AI or using a hopefully aligned AI to generate alignment research has a lot that can go wrong.

We don’t know the order that capabilities will emerge in, so by the time AI is human-level at the tasks we want to use it for, it may be superhuman at tasks that allow it to escape control. Even if we could assume it would perform all tasks at the same intelligence level, that could leapfrog human level in a single model iteration. We don’t know how well we will be able to supervise or constrain future AI systems, and we don’t necessarily know how to check alignment research to see if it’s valid. Even if we’re implementing control measures and using AI to generate alignment research, and especially if we’re simultaneously deploying AI to create real world value, there’s a decent chance that AI will be able to escape our clutches to pursue its own goals.

Overall, if the reason you’re not worried about existential risk from AI is because you expect a specific safety proposal to succeed, I’d say you’re pretty firmly already on team alignment. Doubly so if that proposal is AI control and/or having AIs generate alignment progress, which are clearly not fully fleshed out at this point and could use more resources aimed at their development. Even if you think they’re more likely than not to succeed, investing in those approaches to squeeze out a few more percentage points towards success have an enormous benefit to cost ratio.

“Sure, explosive growth is possible, but cooperating with humanity will lead to faster growth than takeover ”

The models covered in this post are for a single economy. That is enough to show that an AI could exist in autarky, and from there grow sufficiently powerful to take over. What these papers don’t model is whether an AI system would choose to exist in autarky. It may be able to grow faster through economic exchange with the rest of humanity. If this leads to rapid growth for the rest of humanity as well, AI may never develop the technological edge needed to take over. Then it becomes a question of the tradeoff it faces between growing faster initially, and being eventually able to seize all resources.

This is a tempting objection for economists, because historically trade has often been more profitable than violence. Modelling this dynamic is an area for future work, but there are a couple reasons why I’m pessimistic.

For starters, the relative technological progress of AI and humanity would have to be consistent. If it’s jumpy then there may well still be some point during a rapid increase where AI has enough of an edge to take over. Additionally, while economic models usually do not distinguish between technologies, it may well be possible for an AI to have an economic partnership on most technologies while keeping takeover relevant ones secret. Relatedly, an AI may also be able to take actions that hamper the rest of humanity, in a way that cannot be traced back to it. An extremely high level of verification would therefore be needed to ensure that an AI never develops a large technological edge. Finally, there is the question of the offense-defense balance of future technology. Even if both AI and humanity have access to the same technology, there may be a point where offense is sufficiently favored that AI is able to take over regardless.

If this is the reason why you think that AI doesn’t pose an existential risk, then much like the above objection, I think there’s substantial uncertainty as to whether it will hold. I’d like to see your models, and hear what evidence makes you confident. If you’re wrong, we won’t get clear evidence of that until it’s too late. Unlike the above objection, I’m skeptical that there is much we can do now to increase the chances that an unaligned AI is incentivized to trade with us rather than take over, but I’d be interested in hearing suggestions.

This is an area where I’m planning on playing around with models, and seeing what conditions are necessary for continual cooperation. That work will be coming in a future post.