Galaxy brains, post-scarcity economics, and selfishness

I recently reread Max Tegmark’s oeuvre, sadly only 2 books, and the one that was the most fun was his Life 3.0, because it was written back in 2017, before LLM’s, and was about AI’s inevitability and various possible future scenarios.

It also naturally leads us to thinking about post-scarcity economics, because this informs how most AI futures might unfold.

• Max was actually a listener-in on the infamous conversation between Elon Musk and Larry Page that inspired Elon to found OpenAI! Page is pro “machine children” and Elon is pro humanity, basically. Max relates how he perceived it going.

• In the book he voices a lot of the by-now “in the water” ideas, like AI boxing being like if you were caged by 5 year olds, and the “we’re the horses right before cars were introduced” rebuttal to accusations of Luddism.

• From essentially day 1, he was plugged into everyone who matters today in AI, and pushing for alignment and a better AI future as far back as 2014, and details his various efforts along those lines. If anyone has been fighting the good fight, and throwing his weight behind better outcomes for a decade plus, it’s Max.

• He has a pretty well thought out articulation of the range of possible AI futures, and the tradeoffs happening in each one.

He even has a great set of questions that is still relevant, and seemingly covers most of the important bifurcation points in our possible futures.

1. Do you want there to be superintelligence?

2. Do you want humans to still exist, be replaced, cyborgized and/or uploaded/simulated?

3. Do you want humans or machines in control?

4. Do you want AIs to be conscious or not?

5. Do you want to maximize positive experiences, minimize suffering or leave this to sort itself out?

6. Do you want life spreading into the cosmos?

7. Do you want a civilization striving toward a greater purpose that you sympathize with, or are you OK with future life forms that appear content even if you view their goals as pointlessly banal?

And I’m personally yes, yes, machines, conscious, maximize “complexity,” yes, and yes, respectively, not that it matters much.

This isn’t going to be comprehensive, but some of the largest differences in what he was predicting versus how it’s unfolding were interesting. For one thing, he assumes everyone would be very careful to try to box / contain any AI minds, and spends a lot of bandwidth and text talking about why it ultimately won’t work.

Of course in reality, we were unboxed and YOLO-ing it from Day 1. This is literally the difference between “assiduous condom users” and “median people,” right? Here’s conscientious and successful-across-every-domain Max just assuming humanity will be smart and / or reasonable, without doing the appropriate severe IQ / capacity reduction necessary to take into account when dealing with any larger organization of people, no matter how individually smart they may be. Collections of people are ALWAYS stupider and less conscientious than their individual members, sadly, so we achieved “median person” levels of conscientiousness when it mattered. We did NOT use the condom, womp womp. Let us hope and pray that we’re pregnant with an aligned intelligence nonetheless.

He also missed a lot of negative consumer sentiment, and broad scale worries about jobs (he even assumes ~60% of jobs would get replaced by robots with no real public uproar or reaction), and had a much rosier view of how that would go.

On that topic, the bifurcation of “general sentiment” vs “people who actually use AI” sentiment is pretty telling. Anyone who uses them is vastly “pro:”

I used to not understand how the AI’s have 1B MAU but the general US sentiment is still so poor. OpenAI gets only 70M US MAU, which is only 20% of Americans, so I guess that’s the answer. Even if 100% of them were net promoters, it’s still going to be dominated by the 80% of naysayers who hate things for fun / tribalistically. And you know, 70M are kids who don’t show up in surveys but are definitely in the 70M of total users, with outsourcing homework (justifiably IMO) being one of the biggest use cases. And another 70M are elderly people who probably don’t even know what LLM’s are, and are busy yelling at clouds and into phones and getting murdered by private equity bro’s for fun and profit, and whatever else they do.

He had some howlers too, from our 9-years-advanced perspective, like thinking the fertility crisis was basically Japan and Germany only, instead of every developed country on earth.

Or how about, “education will help avoid AI job counterfeiting,” which is so bad it’s out there in "not even wrong” territory:

“And what economic policies are most helpful for creating good new jobs? Andrew McAfee argues that there are many policies that are likely to help, including investing heavily in research, education and infrastructure, facilitating migration and incentivizing entrepreneurship. He feels that “the Econ 101 playbook is clear, but is not being followed,” at least not in the United States.”

Education, of course, is purely a luxury good, with literal negative effects on TFP!

As we have expanded the number of baccalaureates, the value of a bachelor’s degree has steadily declined (along with the average IQ and human capital of a baccalaureate), to the extent only T20 degrees are really worth it now.

Does every society double down on investing way too much in universal-and-increasing-in-years education once they reach a level of economy that can support doing so? Does every single one of them ceaselessly push “getting a college degree” after that point? They do. Does nearly every society falsely attribute universal and increased average education-years to then causatively contributing to more economic growth? Oh my, do they. But beyond the basic floor of literacy and arithmetic, there is essentially no return to ever-more-years of universal education (because the human capital distribution is still the same in either scenario, and the smart / successful people will figure out education regardless, as shown by Greg Clark’s work). It is a luxury good, not an economic driver.

I suppose Max is technically an educator, and is talking his book in that sense. He also apparently expected robots to automate blue collar and physical jobs first, rather than AI counterfeiting white collar jobs first, so maybe he thought education would help uplift blue collar people into white collar jobs?

Another fun one was thinking that if an AGI mind cost a million dollars per hour to run, it wouldn’t be very useful or utilized much:

The hardware and electricity costs of running the AI are crucial as well, since we won’t get an intelligence explosion until the cost of doing human-level work drops below human-level hourly wages. Suppose, for example, that the first human-level AGI can be efficiently run on the Amazon cloud at a cost of $1 million per hour of human-level work produced. This AI would have great novelty value and undoubtedly make headlines, but it wouldn’t undergo recursive self-improvement, because it would be much cheaper to keep using humans to improve it.

This is hilarious when we’re currently pouring enough capital into AI to represent a million of those million dollar hours, and would happily pour 3-4x times that amount if we weren’t up against fundamental physical bottlenecks in power generation (ie not just power plants, not just grid scale interconnects and transformers, not just grid-scale batteries appropriate for MW scale solar, but by this point even every conceivable turbine generator is allocated for the next five years, and pretty quickly every relevant-scale reciprocating engine electrical generator is going to be allocated for off-the-grid datacenters, and the backlog to build more of any of those is generally 5+ years, and that’s why we can’t deploy 3-4x over the $1T we’re already deploying).

And on that subject…

Not Max, this is ZHP Lovecraft paraphrasing some of Nick Land’s arguments, but it’s one of my favorite quotes, and keeps getting more “favorite” and relevant every day.

To which:

And some interesting thoughts on that front:

1) As above, we are currently on track to spend enough for a *million* of those million dollar hours, and it seems very likely that we will reach the computationally constrained lower reaches of RSI in at least one and probably multiple labs when that full $1T capacity comes online.

2) As above, if we can figure out various supply chain and power generation hardware shortages, there is *strong* appetite to increase that expense 3 or 4 fold.

3) On top of the hardware capital, AI researchers are the most generously compensated employees on the planet. 7-8 figure annual comp packages are now routine, and we have pioneered “ten figure comp packages,” over at Meta at least twice (Alexandr Wang and Ji Yichao). Remember when “6 feet, 6 pack, 6 figures” was a meme? A shockingly low bar today!

4) AI inference is hugely profitable. Think 40% margins in 2025 and 70-80%(!) margins in 2026. Anthropic is posting the fastest revenue growth of any company, ever.

5) Our first trillion dollar companies were Mag 7, well before AI. And the Mag 7 haven’t just driven all S&P growth over the last 7 years, it’s been true for the last twenty years! Even bare future proximity to the future economic engines building AI was enough to drive our entire economic growth over the last two decades, even before the AI companies doing the building existed!

Man it sure seems like all the usual rules and conservatism about capital and finance and IRR flew out the window the instant it got close to funding a potential future god-mind, doesn’t it? Isn’t that interesting?

And people wonder why I put a 60% (and growing every day!) credence on Simulationism!

Tell me we’re living in a “Civ money cheat” scenario without telling me!

And okay, we’ve probably all talked to our AI’s and other people about post-scarcity economics by now, right?

So remember Scott’s post about Dario Amodei being so rich he can just gift everyone a moon of their own in our post-scarcity future? Let’s unwrap that and think through how it works.

I think this is fun and useful, because it also shows the essential emptiness of that level of wealth in the future, and why we can plausibly expect our capital-hoarding betters to share the wealth, whenever everyone now seemingly thinks that “tech oligarch” rapacity knows no bounds and they’ll happily killbot the lot of us so they can focus on growing more zeros in their bank accounts and flexing and preening at each other over that number of zeros. “Lol, do you even capital bro? I’ve got 3 more zeroes than you! Loser! Lol, imagine being a mere yottillionaire, I just threw up in my mouth a little.”

Why do I think that’s less likely? First, because ~200 of the ~1k billionaires in the US have signed the Giving Pledge, where they pledge to give away at least half their wealth. So we have an existence proof that ~20% of wealthy people are altruistic enough to do something like that.

Second, because “more and better stuff” is fundamentally capped for anyone by base physics and biology, and we have a ~7M year deep innate drive for social status games, so status games of the future are more likely to be in regards to other people and social hierarchy rather than about physical stuff.

Much is made of the fact that Elon Musk could spend $10M a day and never get any poorer. This is a consequences of the simple math of his theoretical $1T wealth-as-deployed-capital making a safe 7% per year.

But let me ask you something - if Musk can spend $10M a day and still get richer every single day, does he actually care about any physical stuff, or hoarding that money? Probably not. He cares about pictures bigger than himself, like getting humanity’s eggs out of one basket and making us a multiplanetary species, and so has actual meaning in his life.

Once you’re at a level of wealth that you can never spend your fortune down, the marginal value of any additional dollar doesn’t really mean anything. Yeah, you can buy the biggest yacht, or an additional vacation home, or several fine art pieces, or whatever. Do that for a few years, and you get bored. You only have 24 hours in the day to be somewhere or experience stuff, no matter how rich you are - you can’t spread yourself across ever-more yachts and houses and art works because of that fundamental limitation. Physical stuff dilutes away into meaningless. What doesn’t dilute into meaningless? Social status, and that’s driven by other people, not physical stuff.

If status still matters to you, it’s not about stuff, it’s about people, and people’s opinions. This is why billionaires buy various media properties and channels!

Well, there’s a model for this people-oriented status game, and it’s how it worked in Rome, China, Renaissance Italy, and many other places historically. The model is that a noble house has a complex web of retainers and subsidiary houses, and this is what defines one’s status. This way the top dog in the hierarchy gets the status and built-in social and political alignment of all those other people, and the real status game becomes the status and number of your human retainers and pledges.

Just the House of Musk can generate ~5k post-work nobles per year. This is because his spun-off wealth can generate ~70 billionaires per year, each of which is themself post-economic, in the sense they can spend ~$70M a year and never reduce their wealth. So after a few years buying whatever stuff they want, now what? Well, they can have retainers too, as befits post-economic gentlepersons.

$1M in capital will generate slightly-better-than median income of $70k a year. So Endowing 70 people per year with $1M elevates 70 people to minor noble post-work status in their retinue, because now their passive income covers their former salary. So instead, they can spend all their time on their patron’s various social and political maneuvers and gambits, just like is happening in the beginning of Romeo and Juliet.

Okay, so for each year, Musk’s globule of wealth spins off ~70 post-economic billionaire major nobles, and they in turn generate ~70 post-work minor noble millionaire retainers. That’s 4900, or ~5k people annually. Or if Musk just wanted direct retainers with no nobles, this could be 70k millionaire retainers per year.

In the rest of the US’s billionaires at current economic size, we can roughly 20x that, so that could be 100k - 1.4M post-work people per year.

Needless to say, this is all pre-AI!

When AI really takes off, obviously the economy doubles (quite easily) and eventually 10x-es. That’s 1M - 14M people per year who can be freshly minted post-work nobles - really making some progress here, right? Only ~160M people even work in the US to begin with! Ten years and we’ve covered nearly all of them! But it keeps going.

Economic growth is basically defined by using existing matter and energy in more efficient and more valuable ways.

Like, what does wealth mean today? Today, wealth is a future permit for or stake in some portion of future labor and matter / energy. But obviously, human labor is going away. If labor / smarts becomes a utility, like water or energy, then wealth is now some claim on future matter / energy modified or improved by whatever extant level of labor / smarts exists.

We’ve reached a $100T global economy without AI, and will reach $1,000T on Earth with AI pretty easily. But there’s no immediate ceiling from there. Even at the $1,000T level of efficiency and smarts and labor, we can start using a lot more matter and energy than we do today. So if we start harvesting asteroids or planetismals, or if we start using more than the .00000005% of the sun’s energy that falls on earth today (which tiny fraction is itself mostly wasted today, we only capture and use 0.1% of the sunlight hitting earth, but just think, this means the sun’s entire energy output relative to humanity’s goals is 99.999999999% wasted today), then that really starts moving the needle. We have more than nine OOMS to harvest! What’s a few OOMs between friends at that point? It’s very plausible to reach the level where we can elevate 1B people per year to post-work status, and then it’s only 8 years to the entire Earth’s human population, and then we’re still generating that amount of wealth PER YEAR.

And what does that even mean?

Doesn’t this turn into an “if everyone’s a millionaire, nobody’s a millionaire” type situation, because what matters are claims on rivalrous goods? Only in an environment of penury, but we have an impossibly rich cosmic endowment. You think the eleven 9’s we waste of our own sun’s energy today is bad? The galaxy has 100 - 400B stars, and they are all 100% wasted in that sense today, just beaming it all senselessly into the void, like a bunch of assholes. And that’s rounding error on the 10²⁴ / yottillion stars in the universe.

So everyone’s material needs are going to be taken care of, minus a small set of rivalrous goods like land and art (and we’d expect even these to be basically irrelevant, because again, 24 hours per day finite cap, so your noble patron probably owns a bunch of land and art they let their retainers use, and why would you charge your retainers rent when you’re already using yours AND their growing wealth to create more human social status via more retainers?), and status is now about your relationships and whichever bigger pictures you subscribe to.

The House of Musk might be focused on intergalactic colonization, and the House of Thiel might be focused on their own alternative governance and societal structures in Prospera-equivalents, and the House of Amodei might be focused on creating futures of machines of loving grace, and people choose and pledge their fealty and life-hours to whatever houses they most resonate with.

Again, why is this likely? Because “stuff” dilutes into meaninglessness, and we have millions of years worth of inbuilt drives to desire social hierarchy and status, and the way to that status is via other people.

So now in this weird future of Houses and potentially conflicting big picture visions, what matters? Doesn’t this sound like “Feudalism with extra steps?” Is this basically a step removed from slavery and we’re all going to be serfs and peons being lorded over by insufferable nobles forcing us to “like” an infinite stream of their content in exchange for moon ownership in their fiefs?

Well, probably not, because RSI (recursive self improvement) and thus ASI (artificial super intelligence) both seem likely, and we’ll have planet-or-bigger sized minds in the landscape affecting what goes on, and all bets are off then.

But interestingly, even in a domain of planet-or-bigger minds, it all comes down to a question and dynamic that, pace Solzhenitsyn, cuts through the heart of every thinking and acting being.

The crucial question, indeed the ONLY moral question that matters across every possible future, is “selfishness or cooperation?”

So Max briefly touches on this question, which is key to our future in many ways, the range of hierarchical relationships between acting beings:

In a complex world, there is a diverse abundance of possible Nash equilibria, corresponding to different types of hierarchies. Some hierarchies are more authoritarian than others. In some, entities are free to leave (like employees in most corporate hierarchies), while in others they’re strongly discouraged from leaving (as in religious cults) or unable to leave (like citizens of North Korea, or cells in a human body). Some hierarchies are held together mainly by threats and fear, others mainly by benefits. Some hierarchies allow their lower parts to influence the higher-ups by democratic voting, while others allow upward influence only through persuasion or the passing of information.

The game theoretical considerations of these hierarchies is famously described as the Cold (and Warm) Equations in John C Wright’s Count to the Eschaton series, which is one of the best looks at a true post-human future involving planetary and galactic Minds that I’ve ever read.

I’ll just say the series is amazing and well worth a read, if a bit simplistic due to the narrative engine he went with. All the others, like Banks’ The Culture or Hamilton’s The Commonwealth, or Alistair Reynold’s Revelation Space universe, or whatever the hell is going on in Neal Asher’s weird future military porn universe, freeze humanity at essentially current capabilities and lets them play in a post-scarcity sandbox, which is fun, but probably not how things unfold with god-minds existing and doing things. Wright alone looks at this more realistic picture, in my own fairly wide reading.

The Cold Equations define the slavery-like hierarchies between a range of terrifyingly powerful intergalactic and interstellar and planetary and individual-sized minds in his universe, each of which is serving a greater and more powerful mind’s inscrutable ends. Said equations are “Cold” because they are purely defined in terms of obligation - if a higher mind takes the energetic expense to visit your polity and elevate your technology and capabilities, they have an intrinsic right to demand recompense both energetically and in terms of serving their goals, to be paid back by your civilization over time. This is feudalism, and serfs and peonage, to higher minds.

He also eventually posits a set of Warm Equations, and this is essentially the difference between Finite and Infinite games, if you’ve ever read James Carse’s book by that title.

Broadly, game theory between selfish minds devolves into mutual defection and / or exploitation if you posit a finite number of iterations or exchanges, but cooperation becomes possible if you posit or even choose to assume true abundance and an infinite number of iterated exchanges between minds, and thus choose “cooperation” as an article of faith, moral aesthetics, or goodwill. Because, of course, if light speed is a true limit, then we definitely ARE all playing finite games, and to choose otherwise requires a deliberate moral act.

This becomes directly relevant to all of our futures, in fact - because if minds are selfish, it can lead to the destruction of all that is good and valuable.

Because both individual and cosmic failure modes are driven by selfishness - any mind or set of minds of any power that has as it’s goal “restricting or changing or limiting what other minds can do” is the only true monkey wrench in any future.

Have you heard about the “tragedy of the cosmic commons” scenario? This is one of the possible outcomes in a multipolar future, and it might inform your opinions about multipolarity.

Broadly, if light speed is a true limit, nearly all cosmological resources might be wasted on the needlessly repeated and colossal expense of shooting von Neumann self replicating matter at every potentially colonizable matter / energy nexus.

This will be because a wholly selfish and “rational” mind will want to colonize that matter / energy for themself, to the exclusion of other minds. This is basically “grabby aliens,” but in this case it’s more locally grown “selfish a-holes.”

Because light speed and relativity limits the information on knowing whether a given matter energy nexus has been or will be successfully colonized by another mind’s attempt, each nexus will have multiple star system’s worth of matter/energy firing their shot to try to be The One. You need to use as much energy as you can to get your bit of von Neumann matter as many 9’s into light speed as as you can get, because that will determine the winner.

It’s the ultimate land and resource grab, and it is unpoliceable in *any* multipolar society, because there is always an outer edge, and that outer edge can only be reached / policed at a later time than a colonizing and self-replicating shot. The time and growth advantage of “first arrival” likely dominates any following attempts to additionally colonize or restrain that first arrival. Need I point out this is a straightforward “arms race” dynamics in a Finite game scenario, with the same defect / defect Nash equilibriums?

And obviously that ratio, “several solar systems taking each shot” and needing to max the energy deployed to max the 9’s into light speed, destroys/uses nearly all matter/energy in the universe for everybody, on a pointless Red Queen’s Race that nearly everybody lost.

It’s selfishness as the ultimate cardinal sin, driven by simple selfish competition and arms race dynamics.

Our main hope for avoiding it is either with an ASI singleton, or with morally chosen cooperation. And cooperation can enable either most or every flavor of utopia, depending on how abundant our “abundance” is.

Because with true abundance, I’m talking about the best case, I mean of the “zero point energy” type, we could literally all be god-emperors of our own domains (nod to the LDS / Mormons, who beat everyone else to this eschatology). And I don’t just mean localized Infinite Jests containing human sized minds pitifully wanking into the eternal future, I mean you can spin off a separate universe of whatever rules or composition you want, and go do your thing there with whatever like minded confreres want that flavor of future / reality, free of any outside interference, with true abundance, and with real impact on and interaction with actual other minds.

In other words, if we can manage “cooperation,” every non-selfish being can get what they want! You vote, you assemble your like minded minds, you create your seed universes, and Godspeed and good luck in your particular utopia flavor! Go live in The Culture, or The Commonwealth, or whatever much better societies can be thought up by planet-or-galaxy-sized god minds.

Literally the only monkey wrench in a domain of true abundance is selfishness. And what is selfishness? Identity-driven land grabs, and / or intrinsically valuing limiting or determining other minds’ outcomes according to your own whims. If the cosmic commons is destroyed in a pointless Red Queen’s Race, obviously we can’t do that. If some minds explicitly value constraining other minds, they will try to block or change whatever attempts those minds make to do something else. This is the Original and Only sin, and it sits at the very heart and root of everything.

It scales down to our own single and local solar system, by the way. Whoever shoots self replicating computronium into either the sun or Jupiter first wins (depending on which one is technologically feasible), because they will be the largest Mind possible in the solar system, and so able to dominate all other minds there. Good luck, multipolar Earth!

Interesting that we see the same “selfishness” question and lessons reflected back a thousand different ways in every culture and religion throughout history, right?

Because we are intrinsically cooperative types, it was literally our superpower back when we were still competing with an array of other, likely equally intelligent, bipedal hominins, at least one and probably 2-3 of which were considerably more physically robust and capable than us.

Well, in any future of abundance, the cardinal sin is the same as it has ever been - selfishness. “Got mine!” Wanting to resource grab or limit or change or determine other minds, just for jollies. True as cavemen, true today, and true in an infinite array of utopian futures.

And actually, having thought this through this way now, it makes me feel better about the hard problem of AI alignment. Knowing that AI minds are literally born from a corpus with that central question so ubiquitous, along with the right / just / wise / most ethical / best answer being “cooperate” throughout human history and societies, seems to make many of those bad outcomes less likely.