How The Enlightenment Ends

“Where were you when I laid the earth’s foundation? Tell me, if you understand.” Job 38:4

Last week, OpenAI announced that one of its general-purpose reasoning models had disproved a conjecture posed by the Hungarian mathematician Paul Erdős in 1946, a problem that had stumped mathematicians for eighty years. It did so by drawing on a branch of mathematics with no obvious connection to the original problem, which is itself remarkable. External mathematicians reviewed the proof independently and confirmed it correct. OpenAI has not yet released the full 100-page document, and the degree of human involvement in guiding the approach hasn’t been fully accounted for. So more details will hopefully follow.

But if the result is what it has been reported to be, this is a significant moment. It represents exactly what AI tools can be used for in advancing mathematics and science and pushing the frontier of knowledge forward. While there are many competing positions on AI, “slow it down,” “speed it up,” “it’s over/under-hyped,” the technology’s application in science, maths, and medicine is where most camps tend to agree it will be most valuable. Everyone is cheering on AlphaFold!

I, too, am most excited about its application in these domains and eagerly await the next AI-assisted Nobel Prize winning moment. But I also wonder whether we can be trusted to use AI to advance our understanding of the world without crowning it something that stands beyond it.

In June 2018, Henry Kissinger (a figure whose legacy remains deeply and rightfully contested) penned an essay in The Atlantic that was widely read in technology and philosophy circles, discussed with interest, and then largely set aside. The title was “How the Enlightenment Ends.” The Enlightenment — the 17th and 18th-century European movement that placed human reason at the center of knowledge and redesigned society around the pursuit of truth and freedom over inherited authority — would be undone, he argued, not by war or sudden collapse, but by a gradual cognitive shift: society abandoning human reason for algorithmic convenience, replacing introspection and empirical analysis with digital outputs, effectively putting human cognition in the passenger seat. “The Enlightenment started with essentially philosophical insights spread by a new technology,” he wrote. “Our period is moving in the opposite direction. It has generated a potentially dominating technology in search of a guiding philosophy.”

In their final collaborative work, Genesis: Artificial Intelligence, Hope, and the Human Spirit (2024), Kissinger, former Google CEO Eric Schmidt, and former Microsoft Chief Research and Strategy Officer Craig Mundie extended the argument into more specific territory. For roughly five centuries, the scientific method has organized human knowledge through transparency: form a hypothesis, design an experiment, publish the methodology, allow others to replicate and verify. The method’s power is not only in the results it produces but in the fact that anyone can, in principle, follow the reasoning. One of the great challenges with our current AI systems is their opacity, sometimes referred to as the black box. We can know what data we have inputted into the system, and we can observe the output, but we cannot fully account for how the system arrived at it. Factually speaking, we do not know why our AI systems work as well as they do. Extend this into the domain of science, and for the first time in five centuries, we could produce real, confirmed results using AI whose reasoning is structurally inaccessible. Not necessarily wrong, just opaque. The authors argue we risk being catalyzed backward into a premodern acceptance of “unexplained authority.” But this time, instead of accepting what could not be questioned because it came from God, humanity might accept what cannot be questioned because it came from a model.

The authors go on to describe a state of “knowledge collapse,” where the intellectual ground humanity has built its understanding on no longer seems to hold. AI’s discoveries could genuinely be life-changing, from extending lifespan to uncovering the origins of life itself, but simultaneously destabilizing without the wherewithal to explain them. Imagine waking up to news that there are actually 50 hours in a day, or there is no time at all? An even greater state of epistemic collapse could occur in trusting the AI’s reasoning when it is in fact wrong.

It is worth being precise about what this scenario actually requires: AI systems far more reliable than those that exist today, and a sustained failure to develop interpretable, explainable methods. Of course, neither outcome is inevitable. There is also a subtler irony worth naming. To argue that AI could produce findings so profound that human minds are simply incapable of keeping up is, simultaneously, to reinforce the very intellectual subservience the argument warns against. It is a warning that doubles as a kind of awe.

Moreover, I don’t know what demonstrates more anthropocentric exceptionalism: to argue we are capable of inventing tools so extraordinary they escape our own understanding, or to argue that human intelligence is the all-knowing, ever-superior force. I struggle to sort through these tensions as I contemplate them.

But that doesn’t mean I don’t see the potential for such an “unquestioned authority” scenario. The instinct to reach for an oracle seems to be a durable feature of human civilization. Moreover, one could argue that the emerging scholarship around cognitive offloading represents the evidential flashing lights on a road to such gradual disempowerment. And an oracle we stop questioning is one whose errors we stop catching. This grows ever more concerning when you account for people leaving marriages because of AI and using ChatGPT as a stand-in for a conversation with God. There are already some who believe that AI is the only hope we have to save humanity from itself.

AI’s presence in science as an on-ramp to unquestioned authority is perhaps even more subtle and conspicuous because it’s cloaked in secularism, and there is no obvious threshold at which concern gets triggered.

I hope this reflection is not taken as a warning against the application of AI in science. Every scientific milestone is a moment of celebration. But instead, as a reminder of the system AI resides within. A technology that exists because of human ingenuity, not outside of it.

Share

Leave a comment