Shaping AI policy as an academic

Sayash Kapoor decided to work on tech policy while working at Facebook as an engineer, watching how it implemented GDPR. What struck him was how a single policy could drastically affect the work of a leading tech company. He went looking for a PhD program that would let him study and influence tech policy. Six years later, he’s shaped a federal report on open-source AI, co-written one of the most widely discussed essays on AI policy in recent memory, and built a reputation in DC that many researchers only acquire well into their careers.

Sayash is the co-author, with Arvind Narayanan, of the book AI Snake Oil and the Substack AI as Normal Technology (along with the widely read essay of the same name). Next fall, he’ll join UC Berkeley as an assistant professor, where he’ll run a lab on the science of AI evaluations. I sat down with him to talk about how he’s built policy influence as a researcher and the takeaways for those interested in doing the same.

We cover:

• Why tech policy is more like venture capital than engineering, and why the long-shot bets are worthwhile

• The different ways research can shape policy

• How to have impact even from institutions that don’t necessarily reward it

• How Sayash picks research questions, and what AI policy questions feel neglected to him right now

• What technical people who want to contribute to policymaking need to learn

Remco Zwetsloot: You wrote an essay about how tech policy “is only frustrating 90% of the time”, which seems about right from my think tank days. I’d love to unpack both the impact and the frustration themes there. But let’s start with your story: how did you end up working in tech policy as a computer science PhD, when most people around you are probably not doing that sort of work?

Sayash Kapoor: Before I started my PhD, I was working as a machine learning engineer at Meta, or Facebook back in the day. While I was there, Facebook was undergoing its GDPR implementation phase. They had just been asked by the European Union to implement a bunch of changes to how the platform operated. You might agree or disagree with what GDPR is or how it’s implemented, but one thing was unmistakable: this single piece of legislation was significantly affecting the day-to-day operations of a hundred-billion-dollar company. This was a company with around 50,000 employees at the time, and in every single team, something like 20% of resources were dedicated to figuring out how to implement it and what the right thing to do was. It made clear how much of a lever policy is for influencing what AI companies, and tech companies more broadly, do.

From that moment I started thinking about a lot of my work from this perspective of leverage. I very quickly realized that being an early-career researcher or software engineer at a tech company was maybe a little lower leverage, at least back then, compared to being able to influence policy outcomes directly. At the same time, I was not naive. It’s very hard to influence policy. So I started looking for PhD programs that could give me the space to develop the expertise to ask and answer questions of tech policy, and also the technical expertise to understand what policymakers are missing and how to fill that gap.

I was very lucky. I often say that getting into a PhD program is as much about luck as it is about your application. It turned out that Arvind Narayanan, my advisor, was hiring that year, that he advertised his position as being about tech policy, and that he was working at the Center for Information Technology Policy at Princeton. All of these stars aligned, I ended up at Princeton, and that’s how I started working on tech policy.

You said “at least back then” when talking about where you could have influence. Has your view of this changed at all?

Sayash Kapoor: One thing that has really changed over the last five years is the presence of non-academic spaces to write and think deeply. I still think academia has an edge over many of these institutions, but CSET is one place, there are folks at places like METR and Epoch and Apollo who are developing AI evaluations, at CDT thinking about the impact of AI on democracy.

Maybe these spaces existed five years ago, but they’ve really sharpened their approach, to the point where someone entering the space could equally well think of themselves as belonging to one of these institutions if they’re thinking about a career in AI or tech policy. They don’t need to commit to a five-year PhD to develop this expertise.

Now, academia does have a comparative advantage. It’s often not enough to think about a topic for six months or a year, which I’d say is the upper end of the time horizon for a project at a nonprofit working on AI policy. In academia you often have the space to work on a single line of intellectual inquiry for two or three years, which is very precious, and we need a lot more of it. But if you’re someone who likes quick-turnaround projects, and thinking about things just deeply enough rather than spending three years on the same question, a lot of these newer institutions can provide a solid space to do that.

Getting back to the idea of “tech policy is only frustrating 90% of the time,” and the impact being worth it. I’d love to hear examples of ways you think your work has had an impact during the PhD.

Sayash Kapoor: There are broadly two ways I’ve seen policy work be impactful, in my own work and in that of colleagues who are academics.

The first is when there’s a specific policy question of interest that your academic research is uniquely positioned to answer. This is rarely the case, but every so often it happens, and when it does, there’s huge and direct leverage on downstream policy outcomes. One example from my own work: a couple of years ago we started thinking about the role of openly released AI models, how they should be adopted, and what risks they pose. A few months after we’d started investigating this, the Biden-Harris administration released an executive order where one of the main provisions was about the risks of such openly released models.

A lot of interesting things had been written about these models, but we felt many of them had missed one important dimension, which seems obvious as an afterthought: when making claims about open models, many of these works did not consider the “marginal risk” posed by them. In other words, they did not consider the risk that already exists in the world without the availability of these models, or perhaps even without AI, and what the fact that these models are released openly adds on top of it.

So we wrote a paper, and then we also wrote something for policymakers to help them reason about how to make good policy on open models. At the same time, the administration had tasked the NTIA with preparing a report on what to do about open models. The report ended up citing our work something like 15 times, and used the term “marginal risk” around 60 times. That’s the kind of impact that happens once every few years at best, if you’re very lucky and the stars align.

A second kind of impact is more diffuse and harder to spot: giving policymakers mental models for how to think about problems. One example is the AI as Normal Technology essay, which I worked on with Arvind. The essay doesn’t make arguments about which policy interventions are right or wrong. It simply tries to communicate an alternative worldview to where many people in Silicon Valley seem to think AI is going.

More than any of the specific proposals in the essay, the impactful thing was helping policymakers first grasp the debate on AI. They had been very confused about the different things they were hearing, whether AI is dooming us to a dystopia, or we’ll have this utopian existence where we cure cancer and solve everything that plagues humanity, and everything in between, including another camp saying this whole AI thing is all fake.

The essay gave them a mental model of the possible outcomes, and of why AI could be thought of as a powerful but still general purpose technology that will follow in the footsteps of previous ones. The policy impact of that type of essay is a lot more diffuse. It isn’t visible in a specific outcome or proposal, but it can still be very helpful, especially for policymakers thinking about the topic deeply.

You’ve talked about how these impact stories make some of the frustrations worth it. How does the frustration show up when you’re doing this kind of work?

Sayash Kapoor: The mental model a lot of researchers have is: we do something concrete and tangible, we put it out in the world, and it has some impact based on the quality of the work. As a rough heuristic, this works in many domains: the more effort you put in, the better your inputs, the higher quality your outputs, the more impactful your work goes on to be.

That’s really not the case in policy. Rather than seeing policy as something where you have a fixed amount of investment and get some fixed return on your time, policy is better seen as taking small and large bets. For a researcher trying to reason about impact, it’s a very different world. It’s almost like going from being an engineer at a big company to being a venture capitalist all of a sudden, someone who has to make these long-horizon, short-odds bets, where every single investment has between a 1 and 10% chance of success, but when the success does come, it is of a much higher magnitude.

The frustration is largely born out of working with a very different mental model of how impact is actually realized. For a new entrant to tech policy, someone just starting to engage with policymakers, this process can be daunting, and it can also seem like a waste of time. Our goal with the essay was to convince that new participant that it’s actually not a waste of time. Even if 90% of the time you don’t have any measurable impact, the impact you do have on the one thing that works out will be far higher than anything else. This has at least been true for us.

Even if 90% of the time you don’t have any measurable impact, the impact that you do have on the one thing that does work out will be far higher than anything else.

We encounter this a ton when advising people trying to break into the space. Some people stick it out, and others bounce off and go back to research or do something else. For example, is it something about people’s “reward functions”, to use an AI analogy? Some people enjoy communicating with audiences that don’t know what they know, and for others that’s no fun at all. Or are there other factors? What predicts who ends up being successful at this sort of policy work?

Sayash Kapoor: A couple of things definitely help. One is institutional support. There’s one world in which you’re a lone researcher trying to give inputs to these massive organizations whose inner workings you don’t actually know. In that world, trying to be impactful in policy as an independent researcher would be extremely lonely and frustrating.

For me personally, the thing that helped a lot was being at an institution where people could tell me: hey, not hearing back from an agency after you submitted a response to their request for comments is the norm. You should not be frustrated. You’re very lucky if they cite you. You’re even luckier if your proposal makes it into the final regulation or legislation text. That’s just not something that usually happens. Having more experienced people showing you the ropes is helpful.

The other helpful part of being at an institution is simply the support for researchers to talk about their work in ways that are relevant for policy. When I was writing my first policy brief, it was really hard to pivot and think about how to communicate these ideas. The “curse of knowledge” is this wonderful term, which Steven Pinker uses in his advice on helping academics write for broader audiences: when you know something, it’s very hard to communicate it to someone who doesn’t, so paradoxically you become a much worse communicator on things you’re an expert on.

We see this all the time with academics using jargon, acronyms, things they take for granted that the audience would also be expert in. It’s really hard to get out of that mindset. So having the institutional support to format things well, to get feedback on what policymakers are looking for, and on which parts of your research are only interesting to you and your peers, is just as important.

Are there other skills you felt yourself having to build as you ramped up your engagement with the policy world?

Sayash Kapoor: One that might seem basic, but honestly is very much needed, is just an understanding of how policy actually works. You’d be surprised at the number of conversations I’ve had with leading AI researchers who struggle to articulate the difference between regulation and legislation. I’ve often thought about writing a simple explainer for how researchers can talk to policymakers.

Fortunately, given the institutions we’ve talked about, and programs like Horizon and TechCongress and the AAAS fellowships that systematically upskill PhD students in policymaking, there’s now more systematic expertise drawing from people with technical backgrounds and translating that into what’s needed in policymaking. But I wouldn’t undersell that barrier. At the end of the day, it’s a very different expertise, a very different skill set that you actively need to choose to invest your time in.

When we talked to Neil Chilson, who also came from a computer science background, he talked about how the engineering mindset applied to policy can go awry, especially if you ignore incentives and the ways your ideas may be interpreted differently by others. Have you found places where the engineering mindset has been less helpful in policy?

Sayash Kapoor: Yes, definitely. I think it goes back to our discussion of reward functions. A lot of us as technical researchers, especially AI researchers, are trained to think of our time in very clear, actionable slots. We can get specific returns on our time investments, we can forecast them, we know how institutions value them.

Within AI policy there’s a very different value system, a very different reward function, where you not only need to figure out what the right answer is, so to speak, but also whether this answer would appeal to specific groups of policymakers. Whether it fits within their broader vision and their promises to their communities and constituencies. Whether it fits their vision for how they’d like to position themselves within the AI debate. That’s a set of questions that technical people might not have pondered very deeply before.

While doing all this policy engagement, you were also doing your PhD, learning these skills, and writing a book. How were you fitting it all in? For people who want to follow in your footsteps, what are the takeaways?

Sayash Kapoor: One takeaway is: for a lot of technical researchers, the unit of output is often seen as a paper. You work on a specific research problem, it ends up in a publication, and it may go on to be impactful, but at the very least you have a peer-reviewed paper at the end. I increasingly think papers as a unit of output are almost meaningless, in the sense that defining research projects and their success in terms of publications leads to short-sighted choices about what to pursue, and it cuts out a lot of the impact that should proceed from scientific research but currently isn’t being realized.

The alternative framing, and this is something Omar Khattab at MIT has written a lot about, is to work on projects, not papers, and to think of papers as byproducts of longer intellectual inquiries you’re spending a lot of your time on.

That’s advice for researchers broadly, but the way it applies to AI policy is that over the course of my PhD, I’ve tried to systematically pick projects that have some bearing on answering policy questions relevant to AI. The work on open foundation models was one long project we worked on over two years. It led to a few paper outputs, but the main output was the development of those intellectual ideas.

It’s a hard shift, especially within computer science, where writing papers is seen as a natural part of doing a PhD, and where academic promotions, tenure, the entire career ladder is built on the presumption that writing more papers means you’re more effective.

So there’s an individual question: how much latitude do you have to change your own reward functions? Even if you’re working within an institution that really values papers, can you reorient yourself to projects that matter, to questions that are actually meaningful? And there’s an institutional question: how do we build scientific institutions so that career progression isn’t at odds with real-world impact?

Jumping off of that: do you feel like your policy engagement was rewarded? It sounds like Arvind was looking for someone doing policy engagement, but in the wider institutional and academic community, was this seen as positive, or did colleagues react with confusion to how you were spending your time?

Sayash Kapoor: That’s largely a function of how you define my set of colleagues. If you mean the people I interact with on a daily basis, whose work I respect, who are genuinely trying to change the world on questions of AI policy, many of them do see it as relevant and impactful, and that means a huge deal to me. That’s what I consider my intellectual community: people who are trying to think deeply about technological transformations and how to improve the relationship between technology and society.

Within computer science itself, I’d say it’s a mixed bag. Parts of it are still very slow to adapt to the pressing need for technical research to respond to questions of policy. There’s a growing apparatus of institutions, and growing acknowledgement that questions of policy and of responding to society are important, but I don’t think that’s been wholesale accepted as the status quo yet. It’s still something most of the CS community is waking up to.

I’ve seen a lot of academics who do a project and then hope at the end that it’s policy relevant, maybe writing an op-ed on the back end. Another approach is to think about policy relevance on the front end: in thinking about what to focus on in the first place, how to research that question, etc. There’s usually a lot more leverage there. It sounds like you picked your projects with policy relevance in mind on the front end. How were you on the lookout for policy demand signals for your work, especially at an academic institution where those signals may not be present?

Sayash Kapoor: I probably think about it a little differently. I don’t think there needs to be an explicit demand signal. In fact, if there is an explicit demand signal, it’s quite likely this is a problem the community has already acknowledged. But putting new problems on the map is half the work of new intellectual pursuits.

The way I often think about it is: I have a mental model of what the AI conversation looks like, what people think the world is or how it functions. It could be something as broad as “most policymakers don’t have good frameworks for thinking about AI and are responding to companies’ frameworks rather than thinking from first principles,” or something as specific as “most people think open foundation models will pose risks to biosecurity.” And then there’s the mental model I have for myself, of how I think the world functions.

Every so often I realize there’s a big gap between the two. I think of AI as a general purpose technology that won’t lead to superintelligence, for example, or I think of open models as not being as impactful to biosecurity, at least in their current form, as some people have argued. Once this gulf widens enough, that’s the opportunity I’m on the lookout for. It usually means either I’m very wrong about the world, in which case, great, it’s a chance to fix some of my misconceptions, or many people are thinking about the world in an incorrect way, or could improve their decision-making on a particular set of problems.

This has also tended to be very helpful for policy writing, simply because if the questions you’re interested in have some real-world implication, it’s likely policymakers are already, or will soon be, thinking about them. When we began working on open foundation models, the executive order hadn’t yet been announced. We organized a workshop on the risks and benefits of open models a month or two before the executive order came out. The folks at NTIA told us that in the wake of the executive order, it was one of the only publicly available resources they found to study up on open models. That’s the kind of impact that’s impossible to realize if you wait for a demand signal.

Is there an actionable takeaway there? Should people carve out time to think fundamentally about their world models and how they might differ from others thinking about AI policy?

Sayash Kapoor: There’s a tendency within the AI community in particular of herding around specific paradigms. There’s an accepted Overton window for ideas, an accepted set of things, whether on the policy front or the technical front, and the community very quickly herds around the consensus opinion. So maybe one takeaway is that there’s a lot of alpha in thinking through things in a first-principles way, not taking the accepted community consensus as given, and being willing to be bold about challenging the consensus when you think it’s wrong. I do think that has a lot of value. But the stakes are a little higher: the burden of evidence is on you when you’re doing that, so it also requires a deep commitment to excellence. That said, a lot of people have these qualities already, and this can be a useful way to pick problems to work on.

There’s a lot of alpha in thinking through things in a first-principles way, not taking the accepted community consensus as given, and being willing to be bold about challenging the consensus when you think it’s wrong.

I’m curious if you’d point out some gaps in the work you think is most missing. A thought experiment I sometimes enjoy: if you could clone yourself and massively increase the amount of work you did, what would you allocate that new capacity to?

Sayash Kapoor: The best way to answer is with the thing I’m spending a lot of my own time on: how do we build a world that’s safe enough to deploy AI? We’ve seen a lot of questions about AI being safe enough to deploy, and the presumption that comes with a lot of them, including conversations at the federal level on what models should be released or restricted, is that we have some ability to control the flow of advanced AI.

My opinion is that this is exactly the wrong way to go about it. I don’t think we can limit the availability of advanced AI, and I especially don’t think we should be putting all of our eggs in the basket of nonproliferation, broadly speaking. So what remains is understanding how we can make society resilient to AI, if or when advanced AI becomes extremely broadly available.

A lot of companies have spoken about the importance of “resilience”, but often still from the perspective of a world where the leading model providers continue to set the bounds on what capabilities are publicly available and what safeguards are on those models. I don’t think that’s going to stay true.

So the thing that’s top of mind for me is: what is the vision of resilience in a world where advanced AI is extremely abundant, available to everyone, with little to no safeguards on the models being made available? How can we safeguard society in that world?

What does work on that actually look like? What would be high-priority projects in a “resilience” research and policy agenda?

Sayash Kapoor: I would hope my clones can help me figure that out. It’s a hard question. Of course, there are things we can do about the “acute” risks. We know AI poses risks like cybersecurity and biosecurity, and to reduce these risks, we can deploy AI models to defenders, we can figure out how to incorporate AI into the biological screening pipeline. And to be clear, all of those risks are important to address.

But I’m perhaps as concerned about the “diffuse” risks of AI as I am about the acute ones, and they’re often not given as much prominence as they should be. One example is the impact of AI on institutions and on democracy. When people have tried to think about AI’s impact on democracy, it’s been through the lens of acute risks. There was this fear of deepfakes being everywhere in the 2024 elections. There were 60 elections worldwide in 2024, and we looked at every deepfake launched into that political process, and found they were no more rampant and no more effective than the so-called “cheap fakes”, which use low-quality editing or even video game footage to mislead people. So that acute risk to democracy didn’t materialize.

But the diffuse risks are much more likely to materialize: we’re on a path where a lot of people will no longer trust local journalism, or any journalism. We’re on a path where we’re increasingly eroding institutions’ ability to function well, within government and outside of it. These are not risks that will materialize in a flash. They’ll be eroded systematically and slowly over time, and we need to think about them as carefully as we’re thinking about the acute risks.

You’re about to wrap up your PhD. What’s next for you?

Sayash Kapoor: I’m joining UC Berkeley in Fall 2027 as an assistant professor to set up my lab on the science of AI evaluations. I’ll continue to work on technical research that can inform AI policy, and expand the scope beyond the kinds of questions I’ve investigated so far, and also make the work deeper when it comes to actually implementing the things that come out of the lab in the real world.

I’ll also be starting something new this fall with Rishi Bommasani and Arvind Narayanan: more on this coming soon. I like to think we’re in a kind of an institutional renaissance, where people are trying new kinds of institutional models and figuring out what the institution of the future looks like. Organizations like IFP, for example, have been very successful at porting that to the government to some extent, making the government excited about new models for how to do science, for instance with the X-Labs proposal. We want to take on some bets that are bolder than what we might have been able to had we not been in this transition phase.

We usually end by asking: especially for those readers who are technical like you but thinking about getting into AI policy, any final words of inspiration or advice?

Sayash Kapoor: The best piece of advice is to have really broad conceptions of what impact means. Within academia, we’re trained to think of impact as impact within our intellectual community. It’s nice to see someone build on your work, it’s heartwarming to see it advance the scope of scientific research. But broadening that view can lead to many more productive avenues for realizing impact from your work.

Just as one example, most academics would not really count citations in policy work as impactful. Policy citations aren’t even tracked on the leading bibliometric sites like Google Scholar. But for me personally, those have been some of the most treasured types of impact for my work.

But this isn’t just a question of measurement. It really requires changing your views on what you consider your work to be achieving. Once you’ve done that, a lot of the actions just follow through. You start treasuring connections to policy. You change the types of questions you’re asking. The stakes of your work are higher. All of this has been really helpful for me personally.

Big thanks to Sayash for this conversation! If you enjoyed this and want to learn more:

• Our guide to PhDs for policy work on emergingtechpolicy.org

• Sayash Kapoor and Arvind Narayanan’s Substack AI as Normal Technology, the AI as Normal Technology essay, and their book AI Snake Oil