Staring into The Singularity

Generative AI has its uses and we could just have integrated it into our lives like other useful things such as electric bikes or the point-and-click interface. The thing that has got the world obsessing about it is the air of boundlessness that surrounds it, which has several dimensions. Some companies see it as a potentially existential threat, which is why they keep ramming forms of it that degrade the user experience down our throats; they want to be sure no competitor has a way to disrupt them with AI. A similar dynamic operates in strategic circles and lies behind the UK’s commitment to a sovereign AI capability. Some companies see it as a way to generate growth in sales that is off their conventional charts – or at least to hype their valuations to new levels while the cloud of uncertainty as to generative AI’s limits lingers. But most fundamentally, there is The Singularity.

The first unambiguous articulation of the idea that we call The Singularity today came in 1965 in Speculations Concerning the First Ultraintelligent Machine. The article was written at the Atlas Computer Laboratory at Harwell by Irving Good, a mathematician who had grown up in the Jewish East End and worked with Alan Turing at Bletchley Park. Good wrote:

“Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make.”

This has the critical ingredient of the machines bootstrapping themselves to construct a turning point in the history of the world. Also “Artificial Super Intelligence” (a step beyond “Artificial General Intelligence”) is pretty much a synonym for “ultraintelligent machine” and “The Singularity” as used today in the context of AI is pretty much a synonym for “intelligence explosion”.

In mathematics, singularities exist as points where the theory breaks down; you can’t for example divide by zero.

When a singularity arises in physical modelling of the real world, it indicates the breakdown of the model; one cannot in any scientific sense talk about what happens at or after a singularity. To go there is to enter a realm of magic, bluster, hype and confusion. And indeed the article that popularised the term, The Coming Technological Singularity: How to Survive in the Post-Human Era, was written by a scifi novelist, published in 1993, and predicted the end of humanity within 30 years. Despite this, through endless repetition and like intelligent aliens, The Singularity has become a cultural fact entirely divorced from evidence. Thus we see that Good’s article is not only the original, it has the virtue of being more straightforwardly expressed than is usual today.

Even so, we see in Good two of the problems that dog the idea of The Singularity to this day: i) intelligence is conceived as a property of individuals; and ii) even within this confine, what constitutes intelligence is left vague.

To many, me included, intelligence is as much a social characteristic as an individual one. For example, isn’t the articulation of new and useful concepts a critical element of intelligence? And isn’t language itself the most obvious and useful embodiment of such concepts? Even if you insist on looking only at individuals, a reasonable case can be made for the explanatory dominance of a social understanding. For example, pretty much everyone today is smarter than everyone 100 years ago in that, with the help of a textbook or video, we can all do the things they could do but they couldn’t do many of the things we can do. We don’t need to ponder the possibility of “superhuman” intelligence in future because, by the standards of any past generation, we already are superhuman.

Thus, read it carefully and you will find a gap in Good’s reasoning; his ultraintelligent machine might be smarter than any one person but that doesn’t necessarily make it smarter than all of us put together. In addition, there is nothing to stop us incorporating its findings into our textbooks, enlarging the shared realm of social intelligence. And the machine doesn’t necessarily help with many outstanding questions. For example, the conundrums of dark matter and dark energy – do they even exist? – do not arise from a lack of intelligence; they arise from a lack of evidence. That is, The Singularity is not a well-defined thing and there are versions of it that would fall short of being a world historical transformation.

Still, even with all these caveats, Good was right that an ultraintelligent machine giving rise to an intelligence explosion must be considered theoretically possible. And it now comes in two flavours: the utopian scenario of abundance in which the machines do all the work and the dystopian scenario of Skynet from the Terminator movies in which the machines wipe us out. This is how we end up with charts like this one from the Federal Reserve Bank of Dallas, one of the 12 regional elements of the Federal Reserve.

The red line shooting up is the utopian scenario and the purple line shooting down is the dystopian one. The orange and green ones, almost indistinguishable, are the business-as-usual scenario and the scenario with the boost estimated from generative AI built in. Magical thinking now co-exists with sober economics in the central bank of the US, not to mention the pages of the FT.

We have ended up with a riddle wrapped in a mystery inside an enigma. The riddle is the nature and function of intelligence in our society, and hence the form The Singularity must take to truly transform it. The mystery is whether generative AI is capable of yielding such a world historical Singularity. And the enigma is whether this would be good or bad. Our minds move round and round from riddle to mystery to enigma and back again, a cycle of bewilderment that makes it easy to hype the value of AI products and firms.

Now, many arguments have been put forward to contest the claim that generative AI has put us on the path to a Singularity of any kind. Some are qualitative – what can you expect of a stochastic parrot? And some are quantitative – for example the energetic one.

The human brain is incredibly energy efficient compared to any kind of artificial neural network. The brain operates on the same power as a light bulb (12 watts) while an AI of comparable complexity would require the power of a small city (2,700,000,000 watts). Thus, even if it were technically feasible, Good’s idea that an ultraintelligence would be the “last invention” evaporates because it would be far too expensive for most uses.

Beyond demolishing the idea that generative AI is leading us to either utopia or dystopia, both qualitative and quantitative critiques are helpful in circumscribing what we can expect from it. So, the more the better. And so here I want to develop a historical skepticism that roots analysis of the possibility of a world-historical Singularity from AI in what we already know about how the world has changed and is changing still.

Sometimes, rather than Good, the idea of The Singularity is traced to John Von Neumann. The justification for this is slim, consisting of the following paragraph from a posthumous tribute from a friend:

“One conversation centered on the ever accelerating progress of technology and changes in the mode of human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, can not continue.”

Unknot this and it is clear that Von Neumann did not anticipate Good. The text states that we are approaching “some essential singularity”. However, as we understand The Singularity today, as the initiation of a runaway process, it is in fact hypothesising that The Singularity has already happened. What is being foreshadowed is merely an ominous consequence. And, in a sense, this is exactly what is going on.

If one considers the whole history of human civilisation then there is an inflection point in Europe around 1700-1800 when trends that had advanced slowly, if at all, over centuries start to accelerate. This inflection point marks the advent of what historians call modernity.

Modernity can be understood as a combination of revolutions that happened at roughly the same time, released us from a previous constraint and hence define our world. It can be seen as both a good and bad thing, a shift that sets us free, or one that burdens and undermines us. Thus the best word for the de-constraining seems to me to be untethering since both an animal and a boat may be tethered. Untether an animal and it is freed to move with power and purpose; untether a boat and it drifts aimlessly until it is lost. Both the positive and negative valences of modernity are in it.

Foundationally, modernity can be understood as a combination of three untetherings.

The first is the industrial revolution. What we can make and do is always constrained by our energy sources and before this they were limited to human and animal labour, water and wind. Expanding populations meant we were always teetering on the edge of subsistence and famine. The steam engine, the fruit of an accelerating interplay of what we now call science and technology, broke this tether. As the first independently-powered machine, it made power abundant, controllable and portable – opening the path to production lines that are free of nature’s cycles. For most of the world, famine became the forgotten fear. And we are still using ever more energy to create ever more elaborate machines.

The second is a revolution in ideas about how the human world should be ordered. We used to be ruled by tradition, religion and force of arms or, to put it another way, convention, devotion and obedience. Any expectation that we should govern ourselves, of democracy, was generally missing. But this tether frayed over centuries until it finally snapped in the French Revolution. Since then, we have been arguing over the right way to govern ourselves, and even if tradition or religion are invoked, as they sometimes are, it is not the same because they would be choices, not facts of life.

The third is a mathematical revolution arising from the adoption of algebra, a commitment to rigour, the development of new fields such as calculus and probability, and a desire to apply mathematics to the real world. As argued by many historians, it is this advance in mathematical thinking that led to the scientific revolution. As David Wootton puts it in The Invention of Science, “A basic description of the Scientific Revolution is to say that it represented a successful revolt of the mathematicians against the authority of the philosophers, and of both against the authority of the theologians”.

However, the mathematical effect is not limited to the generation of science and engineering. Rather, there emerges a calculating paradigm that offers an alternative to words for discerning patterns in nature and through which all natural questions might be tackled. The pervasive adoption of this paradigm is reflected in the emergence of new quantitative professions including navigation, surveying, ballistics, accounting, insurance, statistics and economics. Bit by bit the pervasive tether of hesitation arising from uncertainty began to fray; English speakers even stopped using the language of uncertainty, abandoning the subjunctive mood (I would go to America) for the indicative (I will go to America).

And yet – still more! – the mathematical transformation was not limited to how humans relate to the material world but also extended to how we relate to each other. We also got a revolution in how the human world is ordered that seems to advance regardless of debates about how it should be ordered. This relentless transformation can be seen with the help of the distinction between “substantive” and “formal” rationality articulated by Max Weber at the start of the 20^th Century and the idea of “thick” and “thin” rules more recently set out by Lorraine Daston.

For Daston, the thickest of rules is that of the abbot that heads the monastery. His very life is the rule that guides the novice and adhering to it requires not just judgement but deep introspection. Thick rules are flexible, require interpretation, enable discretion and are upholstered with examples and caveats that are part of the rule. For example, the chef might be told, “Lunch should be seasonal, fresh and allow for varying appetites and diets. Don’t just offer a roast on Sunday – especially in the summer.” Thin rules are rigid, eliminating interpretation and discretion and requiring no upholstering; the thinnest are those of the algorithm.

Often we find ourselves navigating both thick and thin rules at the same time. The Highway Code contains both – “Don’t exceed the speed limit” (thin) and “Follow the Highway Code” (thick) – as do games like chess where a forced mate uses brute force calculation (thin) while positional play uses strategic heuristics (thick).

Weber saw that societies everywhere had been built on substantive relations, an idea that includes the bonds of family, friendship, religion and community; thick rules that demand careful interpretation were the norm. By contrast, a formal relationship relies on calculation and thin rules that require little interpretation. Formal relations include capitalist ones since an employer-employee relationship is mediated by a contract composed of thin rules and the value of the other side is calculated. However, they are not limited to them; a planned economy like the Soviet Union is also formal. Weber saw formal relations replacing substantive ones everywhere and under it all he discerned a version of the calculating paradigm, a drive towards a world in which “one can, in principle, master all things by calculation”. This dynamic is Weber’s “rationalisation thesis” and both neoliberalism and the right accelerationism that has become voguish in Silicon Valley can be understood as movements that take this contemporary fact of life and seek to turn it into a virtue at the expense of the democratic instinct we have acquired.

In the beginning, the heavy lifting of formal rationality was done by human labour yoked together in bureaucracies and markets, the smart institutions of the day. Their rules have gradually been becoming thinner. For example, early bureaucracies had a small number of thick rules that imbued their officers with wide discretion. It made sense to get to know your bank manager or to plead your case with the government inspector. But the rules now are numerous and thin. As we all know from a thousand call centres, very often the person we interact with is nothing more than a human face on an implacable process.

That process is increasingly encoded in software and the human face is often replaced by a web site. In other words, within bureaucracies the heavy lifting is increasingly handed off to stacks of hardware and software based on that paragon of rules and calculation, the silicon chip. Equally, in the form of platforms such as Amazon, formal rationality has advanced again so that the bureaucratic element is vestigial, a legal necessity that puts grit in the gears of an institution better understood as an algorithm than a bureaucracy.

Stacks of hardware and software form the realm we call tech and its emergence marks the point at which two of the revolutions of modernity combined, when formal rationality began to advance through mechanical rather than human effort. For this reason, the emergence of tech is an event of world historical importance.

The three great untetherings in turn generate further revolutions, for example in how society evolves. The key pressure on humanity used to be the natural world but now that relationship has been inverted so that humanity is the key pressure on the natural world and we live in an epoch defined as the Anthropocene, the time of the humans. We have been untethered from nature.

Instead, another revolution, it is ever clearer that, as Samuel Butler put it in The Book of the Machines in 1872, humanity now co-evolves with machines – not in the biological sense of genes but in the sibling senses of how both society and we as individuals develop.

Consider identity, the sense of who we are. In pre-modern times, identity could remain stable across many generations. But as technology advances, it prompts changes in society that invite us to reconfigure our identity; hence, for example, the emergence of the working class, an identity that did not exist in pre-modern times, became prevalent, and has now been in decline for close to a century. Such reconfiguration is taking place faster and faster as evidenced by people swapping occupations and religions faster and faster; today children are taught to expect several careers over their lifetime and invited to choose from a menu of religions. Thanks to the machines, the duration of identity has been getting shorter and shorter.

Now, however, the internet means identity is often reconfigured without a change in society. This is what it means, for example, for someone to be radicalised online. Identity is changing faster again and the flip side of this is that a newly improvised identity can spread rapidly, be it trans or MAGA. This new fluidity can be experienced as good or bad but either way we and the machines are shaping each other in an increasingly intimate embrace.

In short, each of the three untetherings of modernity released society from a previous constraint and so ushered in not merely a different era but an era of endless change. They initiated dynamics that continually chew up the present to make the future. So when Von Neumann connected “the ever accelerating progress of technology” to “changes in the mode of human life” he was noticing modernity, the singularity that has already happened. And when he suggested that “human affairs, as we know them, can not continue” he was accurately describing what it is like to live in modernity.

If you want to read someone who has worked this through in detail, try Hartmut Rosa’s Social Acceleration. If you want to feel the authentic shiver of modernity, look into the unsentimental eyes of the scientist performing the experiment at the centre of this painting from 1768. He is not distracted like the courting couple; he is not pained by the death of the bird, as the empathetic young girls are; he is not pacifying, as the master is. He is both telling us something – This is how it is – and asking us a question – Have you got it, are you ready for this?

In this context, it becomes clear where the fragment of tech that is generative AI fits in. Generative AI makes no difference at all to the advance of formal rationality. This is obviously true of the generation of images or music. What about text? Consider a chatbot. It offers no advance in calculation or the following of thin rules. Indeed, unlike a more straightforward algorithm that always follows a thin rule correctly, it will often make mistakes when trying to follow thin rules. Rather, as far as rules go, its use lies exactly in the realm of thick ones, such as how to interpret ordinary language. When deployed by a bureaucracy, it is no more than a replacement for the human face that previously decorated the algorithm. Thus, given what we already know about the way the world is developing, generative AI is a dead end. It is just not germane in any deep sense to the driving forces of modernity that tech has harnessed. Like the electric light bulb, it is a useful fruit of underlying technologies that more fundamentally propel us forward.

Further, generative AI has emerged from formal rationality via computation within a framework of thin rules. Thus a device, such as generative AI, that is poor at manipulating thin rules, is intrinsically incapable of itself improving on generative AI. Thus it cannot generate any kind of Singularity.

AI boosters will say this is out of date, that the latest technical developments have transformed the character of generative AI. Like Gary Marcus, I don’t think so. At the end of the day:

• It remains a form of high-dimensional classification (something I know something about)

• Progress is plateauing and it looks like both the Large Language Models and the chips they run on are on their way to becoming commodities

• Previous enthusiasts have recanted

• There is no obvious route to profitability for the service providers

• Thanks to its AI bets, Oracle debt has been downgraded and the shares are down 40 per cent

• Well-connected investors like Softbank and Peter Thiel have bailed

• OpenAI may now be looking for a government bailout.

The window of generative AI’s apparent unboundedness is closing and its Singularity is evaporating. That may mean a crash for AI stocks. But the singularity of modernity is still going strong, untethering humanity from yet more constraints and transforming its possibilities all the time.