Tzu Kit Chan: Preparing for AGI: The Age of Superintelligence

All views shared here are the speaker’s own, and do not reflect the views of Atlas Computing or AI Safety HK. Any errors in summarizing the talk are our own.

Imagine a car where twenty engineers are building the engine and one engineer is responsible for the seatbelts, brakes, steering wheel, and airbags combined. Would you drive that car?

Thats the question Tzu poses at the nature of current AI safety research. The 20-to-1 ratio is roughly what holds today inside the five companies driving most of the world’s frontier AI research, where capability researchers outnumber safety researchers by about that margin. Broader counts across all machine learning researchers in the US push the ratio to 100-to-1 or higher. 20 is the conservative number he was most willing to defend on the spot, and that needs to change.

TL;DR. AI is moving faster than the safety work meant to accompany it. Hong Kong, like most of the world, is downstream of decisions made in San Francisco and the larger contest between two superpowers. Tzu thinks this is fixable — aviation and nuclear safety are precedents worth studying — but it won’t fix itself. AISHK exists to grow that conversation here.

If everything else from the talk got forgotten, Tzu wanted three claims to stick. AI is a big deal. AI is improving fast. AI is dangerous by default.

1. AI is a big deal because it will diffuse everywhere within a few years. Finance, healthcare, power grids, materials research, factory farming. The reason is simple: cost goes down, productivity goes up. Almost any business that runs the numbers will run them toward AI.

2. AI is improving fast. A frontier model holds the lead for roughly three months before another lab releases something better. Twelve months on, today’s state-of-the-art is, in Tzu’s words, a fossil. In under two years, frontier models went from struggling with grade-school arithmetic to outperforming the best human mathematicians on competition benchmarks.

3. AI is dangerous by default. Not because the people building it have bad intentions, but because we’re building something extraordinarily powerful without anything close to a proportional investment in the safety equivalent of seatbelts. This isn’t a fringe view. Tzu opened by quoting Stephen Hawking, who wrote in Brief Answers to the Big Questions that the development of full artificial intelligence could spell the end of the human race. Yoshua Bengio, a Turing Award winner, and Geoffrey Hinton, a Nobel laureate often credited as a godfather of deep learning, have both staked their reputations on the same position. The smartest academics, as Tzu put it, don’t throw their reputations around for fun.

Making the case for AI safety, in his framing, doesn’t require believing in extinction-level risk. The mid-spectrum harms are serious enough on their own.

He spent the most time on three. First: sudden, large-scale unemployment that arrives faster than welfare systems can absorb it. UBI comes up as the answer, but the existing research on it is much thinner than people assume.

Second: what happens when powerful AI ends up in technically sophisticated hands. The mental model most policymakers carry is something like “what could a five-year-old do with a chatbot?” Tzu’s reframe was: stop picturing the five-year-old. Start picturing a senior cybersecurity researcher with thirty years of experience, gone rogue, pointing a powerful AI at a financial system or a city’s power grid. The threat surface is large enough to make the public conversation look quaint.

Third: CBRN — chemical, biological, radiological, and nuclear risk. About 100 lab leaks are already reported each year worldwide from facilities handling the most dangerous pathogens. Bioweapons cost roughly a hundred times less per casualty than conventional explosives. AI doesn’t create these risks. It expands what a technically trained malicious actor can already do with them.

As Tzu kept saying, worrying about near-term harms while the bigger ones accelerate is like fretting about the size of the toilet seat while the jet engine is on fire.

Tzu is, by his own description, a techno-optimist. He wants AI to help solve climate change, cure cancer, and fix global food shortages. But that optimism comes with a condition: we don’t get the long-term gains unless we first handle the long-term risks.

He puts it via an aviation analogy. Commercial flight is now the safest mode of transport on earth — about one major incident per 7.9 million flights, roughly twenty times safer than a taxi ride. That number came from decades of engineering work plus a regulatory regime with real teeth: airlines that don’t comply lose their licences. The same combination of technical progress and enforceable rules carried us through the early decades of nuclear power. There’s no fundamental reason it can’t work for AI. Of 192 UN member states, 12 are in the international AI Safety Institute network. The other 180 (around four billion people) aren’t.

In the Q&A, someone raised the familiar objection: past industrial revolutions created new jobs in the end, so why panic now?

Every previous revolution automated some form of labour — physical work, calculation, communication. This one automates intelligence itself, which is the thing we’ve always relied on to invent the new jobs each time. Once intelligence is automatable recursively, that escape valve closes, which is unlike anything we’ve seen before.

There’s also no historical precedent for a less intelligent species directing a more intelligent one. To understand what life looks like on the wrong side of that gap, Tzu suggested: ask a chicken.

One practical question on involvement: where does someone who isn’t an engineer, isn’t American, and doesn’t have a PhD plug in?

Tzu pushed back on the assumption that you need a specific background — he didn’t go to an Ivy League school himself. The four entry points he named: technical research, policy work, communications and journalism, and operations — the unglamorous but organisationally critical work of keeping things running, where experience in consulting or finance translates directly.

His advice is essentially free. Read a lot. Write a lot. Think a lot. By his estimate, 100 hours of careful reading and writing can bring a thoughtful person to the working frontier of AI safety discourse, about a weekend a month for a year.

At AISHK, what we can do over the next six to twelve months is build a critical mass of people in Hong Kong taking AI safety seriously beyond the privacy-and-bias framing that currently dominates here. With that, routes to real influence open up. Without it, no later advocacy gets very far.

If you’re meeting AISHK for the first time through this post, here’s how to follow what we’re doing.

• Subscribe to this Substack for recaps, original writing from the team, and reading recommendations.

• Follow AI Safety Hong Kong on LinkedIn and visit aisafetyhk.org.

• Join us in Hong Kong on May 14th for our reading group and social. Details on LinkedIn and in our WhatsApp community.

• An in-person AGI Strategy course with BlueDot is coming this summer, alongside our second speaker event.

You can follow Tzu’s work on LinkedIn and at tzukitchan.com.

If anything here resonates and you’d like to get involved, reach out through any of the channels above — wherever you are.