I now understand how civilisations die

I believe we have dominated our planet largely because of STEM done well: science, technology, engineering and maths, all working together. Just semi-dwarf wheat and the Haber process may fairly reasonably be claimed to have “saved billions of lives”.1 Things may now be changing. In the US in particular, I can see more than a few worrying signs.

I was just reading about the $533M failure of the General Dynamics factory in Texas, with flaming robot arms wildly smashing things. How can swanky, state-of-the-art artillery shell making equipment behave so badly? Apparently, the Army’s due diligence was trumped by a brisk sales pitch. Let’s not talk about the USS Abraham Lincoln, also in the news.2 This all reminds me of the sort of cultural shift at Boeing that led to their 737 MAX disasters.

The idea behind this post is that deep to our former success is a thread of information that sustains progress across generations. I think this thread is now fraying. If it breaks, then we will struggle to keep our societies running.

Information is data embedded in a rich context. In the past, I’ve shown how people found the cure for scurvy—but then lost it, because they didn’t understand the mechanisms.^⌘ It’s like the difference between knowing how to build, and answering questions like “Where should I build?” This may seem a bit dry and theoretical, so let’s try a practical example.

If a tsunami devastates a coastal district and tens of thousands of people die, then this will scar everyone—and their surviving children and likely their children will remember. But after three generations, people forget.

With the 2011 tsunami that followed the magnitude 9 Tōhoku earthquake, a scant few villages survived unscathed, because their ancestors had erected tsunami stones (see the picture on the right, at the start) warning “Don’t build below this line”—and people honoured these. Elsewhere, they ended up like the city on the left.

Writing things down is the necessary start. But we need more. Let’s explore.

Check out the pre-Biblical Epic of Gilgamesh, that originated in Sumer, southern Mesopotamia, a mere 2100 years BCE. It’s still readable today on archive.org.3 The author talks of Ubara-tutu, the father of Utnapishtim, who was instructed by the god Ea to build a boat to escape a flood. Yeah, that’s where the Biblical myth of Noah came from, many, many centuries later. Some bits were lost in translation.

But let’s go a few hundred years further back. It is said that Ubara-tutu had a son, Šuruppak, who left an instruction manual for his own son. One of his clay tablets is pictured above. This is one of the oldest texts around.4 Most of it is just wise advice, like where to drill wells, and so on. They didn’t have tsunamis in Sumer, but he does give advice on carefully siting your fields.

Next, we jump forward about 4,500 years, to 1990. This was the year—just 3 years before the World Wide Web was opened to the public—that I first got Internet access. I had a primitive PC with a 5¼-inch floppy drive and a very slow modem. This was not just access to data, it was access to information. It was access to entire communities, with a shared culture. World experts would freely help you to solve problems—mostly related to computers, of course.

There were some limitations, but many of these made things better. For example, you couldn’t participate in alt.hackers without sharing a ‘hack’, however lame. Often, these hacks were “life hacks” rather than ways to remove protections from software. Then, ‘hackers’ were creative people who could find solutions, regardless of whether the problem was software-related, hardware-related, or life-related. You encountered some slightly wicked hacks like this one:

ObHowToEarnQuickMoneyHack:
When I was a little younger I brought many Italian 50 L coins from holiday back
home (Netherlands) because they are exactly the same size as the dutch Fl coins
(50 L = Fl 0.08). I once bought a snack from a machine with it. One snack  (I
don’t recall what snack) cost Fl 1.25 but you had to pay Fl 2,= and got back,
together with the snack, Fl 0.75. So it costed 2x Fl 0.08 = Fl 0.16 and got 1
snack plus Fl 0.75 !!

Many hackers also dispensed life advice. Fravia (aka Francesco Vianello) was one of my favourites. He worked on reverse engineering, data mining and a lot more. He popularised the life lessons (and cracking tutorials) of +ORC, the “Old Red Cracker”. Here’s a fairly typical quote from +ORC Lesson 1:

In order to crack successfully you need four basic things:
* A passing knowledge of assembler language (the more you know, the better and quicker you crack)
* Some intuition
* Some help from more experienced cracker
* A non mercantile mind (more about this later)

And then it changed. Yeah, okay, pretty much everyone gave up on assembly language.5 But the Internet culture changed. A lot. People with mercantile minds started trying to sell us shit. Today, you’ll still find a few sharing-type people out there (I hope I’m one) but more and more, they are overwhelmed by the sales pitch of merchants. It’s getting worse, too.

I think we’re drowning in garbage. Data of all sorts are prolific, but information has become diluted. I now find it far more difficult to find worthwhile stuff I’m looking for on the Internet, compared to just ten years ago. The search engine is too busy trying to promote saleable items. A lot of the resources I took for granted are gone, or behind some sort of paywall. Others are just less accessible. And many communities are long gone.

I had to look a bit to find those +ORC tutorials, for example. Some have vanished forever. If I had copies somewhere, they’re gone too. The first 8 inch floppy disks were made by IBM in 1971. Their 5¼-inch replacement was obsolete by the mid 1990s, and the 3½-inch disk was then the thing. Can you read one of these today? The icon has outlived the artefact. 💾 There’s little to suggest that our current storage solutions are more enduring, either. Today’s useful website will be taken over tomorrow by someone with a mercantile mind.

I’ve learnt. These days, if I find something on the Web that’s worth preserving, I keep a copy. But will my electronic copy last more than a few years, even if I “back it up on the cloud?” The Instructions of Šuruppak lasted for thousands of years. Will anything you or I store digitally last one hundredth as long? We have a bigger problem too. Ammit.

Ammit is the Eater of Souls. Give a child homework today, and they will most likely ask a bot to write a piece of the required length. Adults will often take this easy way out too, whether they are using it to write emails, a grant application, or a scientific paper. It’s convenient.

But what are we losing? I can see three things:

1. Skill.

2. Trust.

3. The human thread of continuity.

The first problem is obvious. We’re no longer exercising our mental muscles. Perhaps most vulnerable are the young, who may be more prone to accept the machine blurb without ever learning how to develop basic critical faculties. Consider the above graphic by Benjamin Riley, based on data from a very recent paper by David Strömberg and colleagues. Chinese children today who don’t use AI replicate the performance of those in the past (the blue and green animated curves). Those who use AI are all over the place—but a huge proportion fail to perform, the section in purple Riley calls the “deadweight learning loss”. A disturbing analysis.

But the machines are also doing some awesome things. Take their acquisition of maths skills.

In a recent note on LinkedIn, Stephen Klein discusses how OpenAI announced ten AI-generated “math breakthroughs”, and how OpenAI misled everyone. Lied, really. They stole the key argument in one of the proofs from mathematician Steven Miller without acknowledgement. Other ‘solved’ problems had simply been retrieved from the literature (also unacknowledged) by the Large Language Model (LLM).

So I read a bit more. I encountered a rather distressed blog by Tasmin Chu, who has been talking to other mathematicians. Some of the blurb coming out of OpenAI is bloviation. But some of it is genuine. This is a threat on several fronts.

The obvious threat is mathematicians losing their jobs. But there’s a bigger threat. She frames this as the use of AI being “in tension with [the values of the current mathematical community]”. These values are pretty important:

1. Mathematicians have a solid tradition of humility, and a strong will to correctly attribute the work of others.6 As we’ve already noted, LLMs don’t do this well at all.

2. They admit mistakes, especially when shown that they’re wrong. In contrast, LLMs can belligerently argue about nonsense they’ve generated.

3. They currently promote a culture of knowledge sharing and open access to their work. It is extremely bad form to swipe others’ results. LLMs have the capacity to completely destroy this—especially if some bugger from OpenAI can steal your months of work, plug the ideas into an agentic LLM, and a single day and a few thousand dollars of tokens later, pre-empt what you’ve done.

One way to see this is AI-induced ‘erosion of trust’.

Consider the above graphic from Pew Research. Do you find it worrying? I’d suggest that one of the main reasons why the US is such an unhappy nation is a loss of trust. Clearly this is not just Internet-related, as trust in others has been declining over decades. People in many (but not all) other countries are more trusting. Pretty much everywhere I look, I see people spending more time on the Internet, and being taken in by misinformation and crazy ideas. Now we have glib AIs to make things worse.

You can see the insidious way that an AI can produce something worthwhile—and nevertheless steal and subvert things of greater importance. We can be subtly poisoned against other people.

If anything this may understate the problem. With AI deepfakes, we may think we’re communicating with someone we trust, and be completely misled. I just read a piece about ‘Research Gold’, a company that claimed to offer “100% human” services, but turned out to be 100% AI.

I find this worrying. Human threads are being cut. Perhaps we can look at other civilisations, and ask why they died? Let’s jump back to Sumer once more.

Sumer collapsed completely in about 2000 BCE. Their entire culture was subsequently buried underground until 1869, when the Jewish Assyriologist Jules Oppert lectured on ‘Sumer’, the year after his Assyrian grammar was published. A flurry of activity followed. But why did this culture fade away?

There are lots of explanations. For example, it seems fairly clear that over-irrigation resulted in accumulation of salt in the ground. Wheat doesn’t like salt; so they changed to barley. Then even this failed.

That sounds a bit too glib, doesn’t it? If you read the more complex theories of William R Thompson, he identifies multiple cycles of collapse and resurgence in Sumer, over thousands of years. The problems he invokes are changes in climatic conditions, trade failures, unstable politics, and foreign incursions. Does any of this seem familiar today?

But he adds another idea: “Tainter’s diminishing marginal return theory”. As we deplete resources, we need more complexity to handle complex replacement technologies. Eventually, the burden becomes too much, and the society does a nose dive. This resonates with me, and calls out our increasing dependence on inscrutable complexity. But I think he still missed something big.

Like Tasmin Chu, I’m concerned about cultural damage. I just think it’s far bigger and more widespread than the picture she’s drawn. It’s not only a loss of trust. What really worries me is the failure of long term retention of information.

In the past, we’ve had more solid ways to retain information, and supported this with our culture. That clay tablet at the start is 4500 years old. It persisted and was copied—the equivalent of a bestseller—because it had cultural value. The Book of the Dead has an alternate name: Spells of Coming Forth by Day (rw nw prt m hrw) These spells are 3500 years old. We have paper texts that are hundreds of years old, and vellum that’s older still. The Magna Carta was written on vellum in 1215 CE.

It’s interesting to look back nearly 1000 years and read about Arab mathematicians. There was a hierarchy here. The written maths was subservient to the text explanation, and the text was subservient to the author. To become accredited in a book, you first had to know the text backwards7—but you did this by becoming the student of an authority. They gave you the right framework. When you were proficient, then not only were you “certified in the book”, but now you could teach it too. An unbroken chain of people, conferring insights that transcended the text. I’d suggest this has general applicability, today.

At present, we still have lots of texts, written on actual paper, that document our past successes and failures. Some of these are treasured cultural artefacts—but do they just sit there? For some, paper is now passé. Online texts can however be changed or turned off like that <Snaps fingers!> And that’s the lesser problem.

When we started on the Internet, we enthusiasts believed that the advent of the Web would cause communities to grow around the information we could all share. We actually saw this happening, for a time. It then declined. Perhaps we can still rescue things? At present, we are losing those cultural threads that tie data to information. And here’s a final problem that may even turn out to be worse. We’re losing the books themselves.

Something else has been in the news recently. It appears that AI-centred companies are now quietly sourcing rare books published before the rise of generative AI. They have come to the belated realisation that training AI on large volumes of preceding AI slop is a recipe for failure. And they too have worked out that online sources are unreliable, mutable and evanescent.

It is also clear that AI training has a voracious appetite. As models become larger and larger—now using trillions of parameters—more and more training data are required, especially for concepts that are not often encountered. It turns out that AIs are very poor at learning unusual facts.^⌘ They find them difficult to homogenise.

What are those AI companies doing with the rare books? It seems they are splitting off the spines, digitising the pages, and then pulping what remains.

I find this horrifying in its own right, but I am also tempted to see this as a metaphor for the impact of AI on our culture: a homogenisation of everything that is human into pulp that replaces our memories and is useless for further human development. The consumer of souls. Something that robs our children of the capacity to think, to trust, and even to will. The AI is perceived to do it better—so why even try?

This is surely how societies actually die. A loss of will—and breaking of connections. So where are our hackers when we need them, finding solutions? Where are our tsunami stones? And will we even bother to read them?

If you’re as concerned as I am try this: read the end of Tasmin’s blog, where she has some solutions! You might give her an upvote too. Mathematicians seem to be doing it tough! Even if you didn’t click that link however, you should visit this one:

What will you find there? Top mathematicians, who are as concerned as I am, declare that they have “ a responsibility to ensure the continued flourishing of the discipline”. They also say how this can be done.

They state their values. They sensibly examine the threats to these values. They make 11 recommendations for individual mathematicians, 8 for organisations, and just 4 for policymakers. They also have a note for commercial AI:

We call on collaborations between mathematicians and industry to abide, at minimum, by the standards we expect of our colleagues and that are described throughout this Declaration. Such collaborations must respect the freedom of conscience of employees or contributors to speak openly about corporate policies and priorities.

I think that every discipline and industry that uses AI can benefit from reading this declaration, and asking how it applies to them. Mathematicians are some of the brightest people on the planet. They are concerned.8 Shouldn’t we be too?

The Book of the Dead is otherwise known as The Spells of Coming Forth by Day. We may need something similar if we are to confront Ammit on a solid footing. Preferably with a book in hand—and an understanding of how to use it.

Are we going to Come Forth by Day, or will our souls be weighed, found wanting, and eaten. What do you think?

My 2c, Dr Jo.

^{⌘ This symbol is used to indicate posts where I’ve discussed the flagged topic in more detail.}