Human Judgement and Artificial Systems

There is a misapprehension in the world of AI development about what “intelligence” is and the production of intelligence as progress against verifiable tasks. This misunderstanding could end up undermining our ability to make scientific and technological progress.

The frontier of human knowledge is not a matter of applying rules, developing combinations of existing knowledge, or even gathering of facts. Instead, it is a process of conceptual discovery and invention, exploring, naming, and mapping a terra incognita, where ideas are refined and built and turned into waypoints that can then be further elaborated upon.

This frontier constantly shifts in response to what we have discovered not only outwardly, as we build ever-larger edifices of information in a field, but also inwardly, as our exploration uncovers new gaps in theoretical understanding or adds specific structural supports to be questioned or replaced. There is no fixed shape to knowledge and likewise there is no fixed shape to the frontier1.

Put another way: The fields of human endeavour and discovery are never final. Neither ideas that are used to make progress nor even the specific domains that we have identified (e.g. physics, geology, computational biology) are set in stone.

The job of those working at the true frontier of knowledge is to constantly be open to imagining a new state of things while holding everything that we feel ourselves to know in their hands. Everything is up for grabs. Even our notions of what is considered objective and impartial or our ideas about what can constitute a “fact” have changed dramatically over time.

This activity is fundamentally what separates frontier knowledge from games or simulations. By definition the evaluation criteria in these constructed worlds are pre-determined. When we seek new knowledge the criteria themselves are constantly questionable.

The problem facing our current paradigm of AI research is that the skills to pursue this discovery and invention are unavoidably tacit2. We can always create a system that ‘works’ in a particular domain where we previously relied entirely on judgement. We have done this repeatedly with everything from systems that can approximate and surpass chess intuition to now others that find new idea combinations to develop novel — sometimes profound — mathematics proofs.

But doing this involves by definition codifying an approach into an explicit system (in these cases AI systems3). That codification relies on a tacit knowledge. A set of people determined what could count as a task, what constituted success and what looked like training by taking something within them and articulating it in the world.

Unless you believe the tacit, unarticulated beliefs of the people doing a particular set of work on the frontier are complete and final — the best humanity will ever do — this is not the end of the discovery and invention process. Successful codification will never eliminate the need for the use of tacit knowledge in moving forward the frontier; it will simply change where that frontier is.

Discovery and invention cannot rely on explicit knowledge because the job at the frontier is to transform an unintelligible reality into an intelligible one.

Ironically, this means that the very researchers working on these problems don’t know how it is that they do what they do. They may not even recognise that this description applies to their work.

This creates a significant risk as this process of turning our unarticulated understandings into working codifications is also the set of skills most likely to atrophy as we integrate AI into everything.

A researcher’s tacit judgement (i.e. taste) is something built over an entire career. It relies on often-grinding work slogging through the minutiae across a wide range of the domains relevant to their chosen focus4. By removing this work we will not change what the AI is capable of, but we may stop producing the kind of humans who are able to determine what it even means for the AI to be capable of something. This risk will remain no matter how capable the AI becomes.

You wouldn’t see it at first. People with exceptional judgement that use these tools clearly already go much further much faster though they may be doing this perhaps at the cost of a certain level of stagnation in their own research abilities. Yet, those just entering the career path now may never experience the work that originally trained the tacit judgement of those advanced practitioners. Where will they be twenty years later? A tradition of knowledge can be lost in a single generation.

If this frame is right, there is a very possible dystopia where whole traditions of knowledge are lost because we didn’t understand what it was we were relying on to make progress in them.

You can see inklings of what this could look like with anecdotes from engineers, scientists, and mathematicians that we are already bottlenecked on a cohort of people trained in a world before AI for catching tiny errors or questioning the direction of AI-driven work.

Another intuition pump for this is to consider the domain of writing. Here is a quote from a recent NYT article:

The problem with writing with A.I. is that it’s mentally enfeebling — an escalator toward a result when you really need to make a daily habit of taking the stairs. As it becomes ubiquitous, it undermines not only our individual ability to write but also a society’s collective ability to reason, a culture’s inner capacity to create and everyone’s reason to care.

To overly simplify: I have heard since I was in my early teenage years that “Writing is thinking”. If that is in some way true, then allowing your AI to write for you will make it impossible for you to truly think in the way you could have otherwise.

Of course, how this plays out is not settled by any means. We have been codifying knowledge into systems for thousands of years. It is not fundamentally a check on progress (in fact it would appear it is often quite the opposite). In game-like environments it may also be a massive source of strength. Human chess play is considered to have far surpassed where it was in the era before computers.

But we cannot lose the fundamental posture of enquiry and diligence behind this work. We have successfully navigated this before: There is one kind of accounting tacit knowledge built in a world of written ledgers and another for the world of Excel spreadsheets. But this transition can be extremely messy. When we moved away from the old practices it created a flood of undetected errors and sometimes serious financial failures coupled with the loss of a trained standard of verification and perception that previous accountants had been disciplined by.

It is a risk that this technology exacerbates dramatically. This is partly because of what the technology does, offering an easy off-ramp to stop engaging with a wide range of tasks. Perhaps there is an even greater risk in what the technology is claimed to do.