Tom Cunningham blog
Note I wrote this around 2020, when working at Facebook, trying to reconcile practical decision-making with how economists talk about identification. In a sense thes points are all well known. The idea is treated formally in Imbens (2003), Manski’s identification bounds; Oster (2019). I think Mostly Harmless Econometrics discusses the point informally. Still I found this a useful way of explaining…
This page surveys definitions related to recursive self-improvement. Most of the content was put together by LLM agents, there are validation checks but it’s possible this contains errors. Any corrections or additions would be very welcome, send me an email! Note Inclusion criteria (source of truth) Coined (red diamond) — the publication is the earliest in this reference to use the term in its…
Thanks to Nate Rush, Thomas Kwa, Beth Barnes, Eli Lifland, Chris Ong, Basil Halperin, Tom Houlden, Parker Whitfill, Phil Trammell, & Andy Haupt for comments. An apple-picking model of AI R&D. Many people are talking about how AI is autonomously able to contribute to frontier R&D, yet it’s only picking low-hanging fruit: Andrej Karpathy , Terence Tao , Nathan Lambert , Ryan Greenblatt . In this…
Suppose you overtake 10 times as many runners as overtake you. What can you say about your speed relative to the other runners? For concreteness, suppose you’re looping around the Viveros Coyoacan, everyone is running in the same direction forever, and each person started at a random point. Under reasonable assumptions your relative frequency of passing runners will exaggerate your position in the…
Thanks to Zoë Hitzig & Parker Whitfill, among others, for helpful comments. It’s useful to make a distinction between two types of LLMs: Knowledge-sharing LLMs. Traditionally LLMs have been trained with human judgment as the ground truth, as a consequence they rarely exhibit superhuman performance. Their economic value mainly comes from sharing existing knowledge, and the natural business model is…
A prediction: people will move towards producing documents that are machine-verified. A document will come with a checklist so you can see that it satisfies certain properties, as verified by LLMs: Claude Gemini GPT Factual claims are accurate ✅ ✅ ✅ Logically consistent ✅ ✅ ✅ Central idea is novel ✅ ✅ ✅ The writing is readable ✅ ✅ ✅ If your blog post starts with this checklist I’ll be more likely…
Validation Checks Overall: ⚠️ Warning ✅ [43/43] Cited sources exist in posts/ai.bib (programmatic) ✅ [33/33] Table rows have required fields (programmatic) ✅ [33/33] QMD quotes match posts/ai.bib (programmatic) ✅ [33/33] QMD growth values match posts/ai.bib (programmatic) ⚠️ [41/43] Abstracts present for all cited sources (programmatic) ❌ [25/26] Bib quotes present in local fulltext version…
Thanks to comments from Daniel Björkegren, Andreas Haupt, Philip Trammel, Brent Cohn, Nick Otis, Andrey Fradkin, Jon de Quidt, Joel Becker. This is a long collection of notes about economics & AI, prompted by two excellent workshops I attended in mid-September: the Windfall Trust’s “Economic Scenarios for Transformative AI” and the NBER’s “Workshop on the Economics of Transformative AI” . I had…
I’ve spent a lot of time trying to prove things. With diagrams and algebra, back and forth between clipboard whiteboard blackboard & keyboard. I can’t talk about what it’s like for a good mathematician but I can talk about it for a hack. I can prove true & interesting things occasionally but only after wallowing in it for a long time, and after a half-dozen proofs of things that I later realize…
Here are two nice pieces of Bayesian logic observations that help explain everyday intuitions: When an outcome is the sum of two components then your belief about the contribution of the thinner-tailed component will be first increasing then decreasing in the realization of the outcome. When you observe an outlier in some process, which is the sum of multiple components, then: If the components…
When formalizing tradeoffs our decisions typically get worse before they get better. Those who climb the mountain of efficiency first pass through the valley of confusion. Data scientists are often asked to provide a formula to calculate the expected costs and benefits of a decision but this is hard. There are often many subtleties which we grasp intuitively but do not know how to formalize. For…
In short: 1 1 First version January 23 2023. Thanks to comments and conversations with many, incl. Jon de Quidt, David Atkin, Donald Kenkel, Dmitry Taubinsky, Charlotte Paul, & Ruth Cunningham. The majority of variation in preferences is due to the influence of peers. Peoples’ tastes reflect what they are accustomed to, either from their upbringing or from their contemporaries. Tastes for food,…
This note contains some ideas about hierarchical structure in perception, judgment and decision-making that I haven’t seen explained clearly elsewhere. 1 Summary A model of encapsulated inference (“bloodhound”). Many puzzling phenomena in perception, judgment, and decision-making can be explained if we assume that initial judgments are formed pre-consciously in a way that both (a) incorporates…
Thanks to many comments, esp. Ravi Iyer, Sahar Massachi, Tal Yarkoni, Rafael Burde, Grady Ward, Ines Moreno de Barreda, and Daniel Quigley. Summary It is difficult to anticipate the effect of AI on online communication. AI models have already had big effects on content moderation but they are now starting to have effects on content production, e.g. through generating synthetic spam and deepfakes.…
This document describes five technologies for automated text moderation, each roughly correspond to an historical phase. As a working example we will use the detection of “toxic” comments. In practice many different definitions of “toxic” have been used in the industry, and there are a variety of related concepts, e.g. “hate speech” and “offensive”. (1) Keywords The simplest technology is to…
This material was first presented at MIT CODE 2021. Thanks to Sean Taylor among others for comments. Thinking about tradeoffs? draw an ellipse. When making a tradeoff between two outcomes, and , it’s useful to sketch out what the tradeoff looks like, and an ellipse is often a good first-order approximation. The ellipse helps visualize the most interesting parameter: the tightness , i.e. how much…
Introduction I give a simple Bayesian way of thinking about experiments, and implications for interpretation and extrapolation. Thanks to J. Mark Hou for comments. Setup: The canonical tech problem is to choose a policy to maximize long-run user retention. Because the policy space is high-dimensional it’s not feasible to run experiments on every alternative (there are trillions), instead most of…
Thanks to comments from many, especially Giorgio Martini , Grady Ward, Rob Donnelly , Inés Moreno de Barreda , and Colin Fraser. If we train AIs to imitate humans, will they ever beat humans? AI has caught up to human performance on many benchmarks, largely by learning to predict what humans would do. It seems important to know whether this is a ceiling or we should expect them to shoot out ahead…
A model of internet media: the platform chooses the composition , the user chooses the quantity . I think this is a nice crisp way of modeling how media platforms (FB, YouTube, TikTok) make their decisions about content: they chooses the mix of content, i.e. the shares of each type, and then their users choose the quantity . The platform is choosing the fillings for the sushi roll and the consumer…
TL;DR: The experiments run by Meta during the 2020 elections were not big enough to test the theory that social media has made a substantial contribution to polarization in the US. Nevertheless there are other reasons to doubt it. Summary Thanks to Dean Eckles, Solomon Messing, Jeff Allen, & Brandon Silverman for discussion which led to this post. I put together the spreadsheet summary of results…