Finding a Niche in China x AI Governance

I spent the last ten days of July at ML4Good’s first AI Governance bootcamp in a Château in Saint Nizier d’Azergues, near Lyon.

It’s exciting to take a first serious step into this space, and it’s super cool to take that step with thirty brilliant and lovely people working to navigate the messy business of governing AI.

I’d really recommend the ML4Good bootcamp to anyone interested in building up to a AI policy career! The bootcamps give you a great overview of the AI safety landscape, examples of technical implementations, governance frameworks, and strategy etc. All this while spending a fun10 days in the French countryside, replete with football games, rain-soaked quests to the nearest pub, and hilarious theatre improv sessions.

The instructors were excellent, and the participants ranged from card-carrying Effective Altruists to people who’d barely heard of the movement. I was especially impressed that the organisers managed to break out of the EA bubble and attract a handful of people from government, business, and non-EA nonprofits.

As well as learning a bunch of theoretical and technical governance stuff (based on this excellent AI Safety Atlas), some practical negotiation and networking skills, and insights from people in the policy weeds, it also functioned as a bit of a kick up the backside to just start doing things, which is what I’ll focus on for the rest of the post!

I realised a while ago that I’m reasonably well-placed to contribute to work on AI Governance in China. I’ve got over 5 years of China-related research experience, Chinese fluency, a decent workflow for finding out what’s happening inside China, and I’ve had at least a few toes immersed in the AI Safety space for a few years now. I’m also very motivated (read: terrified) by the cause area, and genuinely excited at the prospect of building serious expertise on some obscure niche in this space.

But where to begin? I guess this has been a roadblock for me so far. When you’re trying to dive into a topic, it can feel like you need to be an expert in everything, from obscure Chinese legal texts to hardware-enabled mechanisms for chip governance. But you’ve got to start somewhere! To work out where, I’ve tried to ask myself:

• What are the bottlenecks?

• What is important, tractable and neglected?

• What are areas where I need a PhD in just to speak about vaguely sensibly, vs. where I’m able to do insightful work from my existing knowledge/general awareness and a short deep-dive?

• What might be interesting for someone casually interested in this space to read?

• What might be essential knowledge for someone actually making big decisions?

• What would actually motivate me to write more, and go deeper?

For the first time since I started following this field, I decided to map out 90 potential intersections between China and AI Governance (spreadsheet here), in hope of finding a few topics that might have both broad enough interest to attract attention to a blog, and have some kind of relevance to people thinking seriously about this.

Quite a few people are already writing in this space, and I’m starting to form a mental map of the organisations and stakeholders clustered around the overlaps between China and the sub-fields of AI governance. From these sources, a few potential niches are already coming into view.

I narrowed down to four topics: compute overhangs, the “arms race” narrative, treaty red-teaming and post-crisis governance.

These are the 4 questions I’ve come up with:

• Might China have a massive compute (or data) overhang for future AI scaling? What are some plausible future scenarios?

  • There are some well-researched blog posts on China’s Weird Chip Surplus. On the surface, it just looks like Chinese companies have a bunch of chips that can’t be used for training. But what happens if we actually hit a new scaling frontier, or we enter a new paradigm for compute or inference scaling? What’s actually going to happen if a bunch of compute becomes useable? How does this change if we enter an “AI endgame” scenario?

• What role does the “arms race” narrative play in ambitious governance scenarios involving China?

  • This question seems kind of “meta”, and pretty interesting. In current political discourse, I think most mention of an “AI arms race” is a distraction, or even a barrier to important regulation in the US. But there are clearly a few governance paths, like an AI moratorium, where US–China competition and/or cooperation becomes the real choke-point—“if we don’t build it, they will” is clearly true in the long run. I want to look at when is it and isn’t useful to talk about “arms races”, and how belief in the existence of an arms race might affect different scenarios.

  • Addition: Just realised that this recent piece was written on this topic—definitely looks worth critiquing.

• Do current ideas for ambitious AI treaties (e.g. A Narrow Path) take seriously how decisions are made, and how international treaties are managed, in China?

  • Okay, this one’s slightly cheating, because I’ve already done an Apart Research hackathon on the topic, and I’m just building it into a blog with their help! This should be coming tomorrow or Tuesday!

• Post-AGI crisis governance: what do we expect to happen, and who’s involved?

  • There’s lots of general writing on “crisis management” in international relations, often borrowing from nuclear analogies. But what about specific AGI-related risks? If we take seriously the possibility of an escape or loss of control scenario, extreme misuse, or an explosion of power-seeking behaviour (either in the US, China or elsewhere), what actually happens in the fateful hours and days that follow? Who tells Xi Jinping? When does control shift from civilian to military? How soon can Xi pick up the red phone to Trump?

For each of these, I’ll be doing:

• A 2–3 day focused deep dive

• Pulling up a few under-read Chinese sources and maybe a couple of careful translations

• A few unfiltered hot takes

In terms of style and approach, I’m going to shoot for:

• Writing relatively candidly and apolitically on each topic.

• Avoiding “normal mode”. Not many people are writing about China x AI like we’re on the road to AGI, intelligence explosions, or existential risk.

This blog will be the home for this over the next few weeks. I hope some of it proves useful, or at least a bit interesting!