Don't just not do bad things. Do good things.
I continue to believe we should pause frontier AI development. Any discussion of alternative strategies should be thought of as planning for contingencies. A unifying driver behind many post-alignment risks —catastrophic risks that remain even if we solve the alignment problem—is that by strong default, ASI would end liberal democracy . Liberalism—in which people have individual rights, autonomy,…
Here’s some cool stuff I’ve enjoyed recently. FloatHeadPhysics is a YouTube channel that breaks down difficult physics concepts in a way that makes them clear. One of my favorites: I finally understood Schrödinger’s cat! The video talks about the double slit experiment, and explains why electrons don’t go through slit 1 or slit 2, but they don’t go through “slit 1 or 2”, nor do they go through…
To achieve the best possible future, we must know what that future looks like . In other words, we need to solve ethics. 1 The problem of solving ethics is so large and abstract that it’s difficult to say useful things about. In lieu of any structured analysis, herein lies a collection of thoughts about the problem. Contents Contents Acting in the face of moral uncertainty Can we discover facts…
I prefer to avoid saying things that I suspect might make me look stupid. Over the last couple of years, however, I’ve made a conscious effort to write publicly about what I’m thinking about, even if I’m afraid it might sound dumb. Sometimes I have this feeling, like: I have an observation about the world, about something that doesn’t seem to add up. Probably I’m missing something, and everyone…
Some people think we should pause AI, but not now. They say we should wait until AI reaches human level, 1 because: It’s not (catastrophically) dangerous until after then. Human-level AI will help us do safety research. Alternatively, other people (like me) think we should pause AI as soon as possible. Katja Grace wrote a nice concise case for pausing ASAP . I have something I’d like to add:…
I continue to believe we should pause frontier AI development. Any discussion of alternative strategies should be thought of as planning for contingencies. “Super-persuasive” AI is dangerous because a misaligned ASI could persuade humans to help it take over. But setting that aside, even if we manage to make ASI friendly, it may provide super-persuasive but mistaken guidance that permanently sets…
An argument goes: If we don’t build aligned artificial superintelligence, we risk driving ourselves extinct for some other reason. We should rush to build ASI quickly, in spite of the risks—the longer we wait, the more vulnerable we are to extinction from a different cause. Other than ASI, the biggest extinction risk is synthetic biology. Some lab could (accidentally or on purpose) develop a…
Say you work at a private company that gives you stock options or RSUs . How should you value your stock? If you have a choice between getting more stock or more cash salary, how do you decide which to get? If you have the chance to sell some stock, should you do it? Stock is risky and inflexible (especially if you work for a private company where you can’t easily sell shares), but you might be…
Prior discussion: niplav’s shortform (2025); Planning for Extreme AI Risks (2025) by Joshua Clymer A frontier AI company (any one, I don’t care which) should close shop and make an announcement along the lines of: Powerful AI could end the human race. We are too worried that we don’t know how to make this technology safe. We have decided to shut down because we don’t want to be responsible for…
(Spoilers in this post are hidden with spoiler tags.) What made Project Hail Mary so good? Among other reasons, it’s because the science drove the story, instead of the other way around. Character-driven stories and hard sci-fi might take up opposite positions in the ancient battle of “people vs. things”; but when they work, they work for fundamentally the same reasons. In mediocre…
Image credit: Jebulon To prevent superintelligent AI from killing everyone, I would like there to be a strong international agreement banning the development of ASI until it can be proven safe. But that sort of agreement requires a lot of political buy-in and coordination. In the meantime, it may be easier to get light-touch AI safety regulations passed. To what extent do weak regulations decrease…
Depth-first plans lay out a path from here to aligned superintelligent AI. We need those kinds of plans. But depth-first plans depend on many assumptions: “We will make AI safe by doing step 1, then step 2, then step 3.” Step 1 only works under condition A, step 2 requires condition B, step 3 requires condition C. If A or B or C is false, the whole plan fails (and there’s a good chance we all…
Three categories of futures, depending on how AI goes: ASI timelines are long. ASI timelines are short, and we’re on track to solving AI alignment. ASI timelines are short, and we’re not on track to solving AI alignment. If we want to make a good future for all sentient beings, each of these futures has different implications for what we should work on. If timelines are long… …we can prioritize…
They say exercise improves sleep quality. Is that true for me? To test this hypothesis, I took my daily calorie expenditures from the Apple Health app and correlated them with that night’s sleep time. 1 I also included caffeine intake as a potential confounding variable. The hypothesis: when I exercise more, I’ll get better rest that night, and therefore wake up earlier. The results: 2 name coef…
A few years back, I got a big pile of money from working at a tech startup. I put a lot of that money into a donor-advised fund. Since now I make hardly any money, that DAF might represent the majority of my lifetime donations. How much of my DAF should I donate per year? In particular, how much should I donate in light of short AI timelines? I created a simple model to answer this question. The…
TLDR: I basically don’t. Contents Contents Ethical concerns Thoughts on how to avoid becoming corrupted Future worlds What happens in the lead-up to ASI? Predictions are hard, especially about markets Trend-following The EA portfolio Leaning my investments in the right direction Appendix: Some specific predictions Notes Ethical concerns If you stand to make money from AI, that incentivizes you to…
I’d guess maybe a 50% chance that we’re all dead within 5–20 years because somebody will build superintelligent AI, and then the superintelligent AI will kill everyone. AI developers are on the way to building smarter-than-human AI. Present-day AI is making rapid progress. We humans can still do plenty of things that the AIs can’t, but AI companies are working hard to change that, and they’re on…
Previously, I wrote about how investors can simulate leverage via concentrated stock selection . That’s still true as far as I can tell. However, I also wrote something that I now believe to be false: concentrated equal-weighted factor portfolios have alpha on top of value-weighted factor portfolios. The numbers I found before were not wrong per se. However: The alpha came primarily from small-cap…
If civilization advances its technological capabilities without advancing its wisdom , we may miss out on most of the potential of the long-term future. Unfortunately, it’s likely that that ASI will have a comparative disadvantage at philosophical problems. You could approximately define philosophy as “the set of problems that are left over after you take all the problems that can be formally…
Even if we solve the AI alignment problem, we still face post-alignment problems , which are all the other existential problems 1 that AI may bring. People have identified various imposing problems that we may need to solve before developing ASI. An incomplete list of topics: misuse ; animal-inclusive AI ; AI welfare ; S-risks from conflict ; gradual disempowerment ; permanent mass unemployment ;…
The existence of liberal democracy—with rule of law, constraints on government power, and enfranchised citizens—relies on a balance of power where individual bad actors can’t do too much damage. Artificial superintelligence (ASI), even if it’s aligned, would end that balance by default. Cross-posted to LessWrong and the EA Forum . It is not a question of who develops ASI. Whether the first ASI is…
Many people in the animal welfare community treat AI as a powerful but normal technology, in the same category as the steam engine or the internet. They talk about how transformative AI will impact factory farming and what it will mean for animal advocacy. Only two futures are plausible: AI progress slows down—either because it hits a natural wall, or because civilization deliberately makes the…
Cross-posted to the EA Forum . An aligned ASI can be “ethical” 1 (it does what we think is right), or it can be corrigible (it does what its principals want). If it’s ethical, that means it will refuse unethical orders, but the tradeoff is that you can’t change its mind if you realize that the AI is wrong about ethics—its values are permanently locked in. 2 Assuming we succeed at aligning ASI to…
Cross-posted to the EA Forum . In the far future, why would people use up precious resources recreating wild-animal suffering, when they could do so many other things with those resources instead? That argument is an important reason to expect aligned ASI to produce a future that’s okay for animals, even if it’s narrowly focused on human welfare and doesn’t care about animals at all. This is an…
Cross-posted to the EA Forum . If transformative AI arrives soon, what interventions might improve animal welfare in the post-TAI world? I came up with a quick list of ideas and wrote some pros/cons for each. These ideas talk about animal welfare, but most of them could also be applied to the welfare of any nonhuman sentient being (e.g. digital minds). I started from the ideas I covered previously…
Cross-posted to the EA Forum . Epistemic status: Speculating with no central thesis. This post is less of an argument and more of a meditation. A decade ago, before there was a visible path to AGI and before AI alignment was a significant research field, I figured the solution to the alignment problem would look something like Coherent Extrapolated Volition . I figured we’d find a way to get the…
Cross-posted to the EA Forum . Last September, I wrote : There’s a (say) 80% chance that an aligned(-to-humans) AI will be good for animals, but that still leaves a 20% chance of a bad outcome. AI-for-animals receives much less than 20% as much funding as AI safety. Cost-effectiveness maybe scales with the inverse of the amount invested. Therefore, AI-for-animals interventions are more…
Cross-posted to the EA Forum . AI alignment is typically defined as the task of aligning artificial superintelligence to human preferences. But non-human animals, future digital minds, and maybe other sorts of beings also have moral worth; ASI ought to care for their interests, too. In broad strokes, if we place all alignment techniques on a spectrum between getting AI to do things that their…
(Or: Why I don’t see how the probability of extinction could be less than 25% on the current trajectory) AI developers are trying to build superintelligent AI. If they succeed, there’s a high risk that the AI will kill everyone . The AI companies know this; they believe they can figure out how to align the AI so that it doesn’t kill us. Maybe we solve the alignment problem before superintelligent…
Introduction Most people who write about AI and investing fall into one of two camps: traditional investors who see the high valuations of AI stocks and say it’s a bubble; 1 or AGI-pilled investors who will buy AI stocks at any price, regardless of fundamentals. There’s only a tiny intersection of people who understand that AGI is not a normal technology while also recognizing that fundamentals…
Value investing had a singularly bad run from 2007 to 2020. (And it hasn’t done great since 2020, either.) Is that because value investing is broken, or did it simply hit a streak of horrendous luck? Skeptics of value investing have made many claims about why value investing doesn’t work anymore, but these claims tend to be light on evidence. 1 Value investing proponents have empirically…
Summary Time series momentum (TSMOM) is an investment strategy that involves buying assets whose prices are trending upward and shorting assets that have a downward trend. In 2012, Moskowitz, Ooi & Pedersen published Time Series Momentum 1 . They analyzed a simple version of the strategy that buys assets with positive 12-month returns and shorts assets with negative 12-month returns. They found…
You may have seen this graph from Chris Olah illustrating a range of views on the difficulty of aligning superintelligent AI: Evan Hubinger, an alignment team lead at Anthropic, says : If the only thing that we have to do to solve alignment is train away easily detectable behavioral issues…then we are very much in the trivial/steam engine world. We could still fail, even in that world—and it’d be…
Many people in AI safety, especially AI policy, want to increase government expertise. For example, they want to place people with AI research experience in relevant positions within government. That may not be a good idea. People who better understand AI can write more useful regulations. However, people with relevant expertise (such as ML researchers) tend to be less in favor of strong…
As of a few days ago, my website supported comments via Commento . If you click on that link, you will find that the page doesn’t load. Unfortunately, that website was also hosting my website’s comments, so all the comments are gone now, and I have no way to recover them. 1 Some of y’all left some good comments, but future readers will never know what they were. 2 (I knew Commento was no longer…
I can’t figure out any good way to put a smiley emoticon inside parentheses. There are five choices, all of which are bad: Do the straightforward thing of just writing it (which puts two parentheses next to each other in a row, and makes it unclear where the smiley face ends and the parenthesis proper begins :)). 1 Do that, but put the period inside the parentheses. (Which requires restructuring…
I published a post every day of November as part of the Inkhaven program, in which we are required to publish a post every day of November. Some of my readers knew that; others were confused about why I suddenly started posting so much. If you’re an email subscriber, you didn’t see every post because I only sent out the good ones—I didn’t want to bombard you with emails if you were accustomed to…
I’ve been interested in lucid dreaming since high school, with just enough success to say that my efforts haven’t been a complete waste of time. I have a lucid dream once every few months, which isn’t great. But I still do reality checks multiple times per day. The simplest way to lucid dream is to follow two steps: Get into the habit of writing down your dreams as soon as you wake up, so you get…
A silly argument: The goal of this activity/institution is to achieve X. It doesn’t really achieve X, but it does achieve Y, which is even better! If achieving Y is more important, why on earth would you go about that by trying and failing to achieve X? You should directly focus on Y instead. School allegedly teaches geometry/history/etc. Sometimes people complain that these skills aren’t useful,…
Not that I’m remotely as funny as George Carlin, or that this list is funny at all. But he had many complaints and grievances , and today I would also like to complain about some stuff. This post contains spoilers for a lot of things. I won’t hide spoilers, but I will say the name of the thing before giving the spoiler. When people in their 30s or 40s (or even 20s) say they’re “living through my…
This meme got me thinking: That feeling when you’re smart enough to know how awkward you are, but not smart enough to know how not to be awkward The reason it works that way is because not being awkward is NP-hard, and I can prove it. It is known that the SAT problem is NP-hard. SAT, or the satisfiability problem, is the problem of taking a logical statement involving a series of boolean values,…
In the spirit of You Can Just Do Things, here are some things I Just Do. Some of them are weird; others are normal, but frequently overlooked. Edited 2025-12-03 to add a ninth thing. You know how on a hot summer night, your pillow gets hot, and then you flip it over to the cool side? But then before long, the cool side becomes too hot? You can fix this by pouring cold water on a towel and then…
I played a lot of MTG from age 9 to 14 or so. I picked up the game again recently and I was immediately better at the game than my 14-year old self. I don’t have any direct way to prove this, but I’m pretty sure it’s true. When I was a kid, I liked zombie cards. (And I still do! 1 ) I looked up zombie decklists online and they all used Carrion Feeder . As a kid, this confused me. Carrion Feeder is…
Nearly all gaming keyboards use the conventional typewriter-inspired keyboard shape. That is not a good shape for typing or gaming or frankly anything else. image source The gaming experience is vastly improved by keyboards that have thumb keys, such as the Kinesis Advantage or the Maltron . If you move the Shift key to one of the thumb keys (which you should 1 ), then Shift-hotkeys and…
A few years ago, there was some publicity around a Navy fighter pilot who claimed to have seen an unidentified object that couldn’t possibly be explained except as an alien phenomenon. Many people considered this to be indisputable proof of aliens. “The pilot is an expert, there’s no way he could have been wrong.” I am much more willing to believe that someone can make a mistake, regardless of how…
This post contains spoilers for the first episode of Pluribus. Pluribus is the new show brought to you by the same team who made Breaking Bad and Better Call Saul. The creators are all at the peak of their craft, including the writers. The show has only just started airing, but I’m watching it with confidence because I trust them. I’ve listened to every episode of the Breaking Bad and Better Call…
A reborrowed word is a loan word that goes from language A to language B and then back to language A. I think they’re neat. A classic example is pidgin . A pidgin is a grammatically simple proto-language that emerges when two groups from different places have to learn to communicate. The word pidgin originally described a simplified form of English spoken by Chinese business people, with pidgin…
The ethical principles that most people hold—and hold most strongly—go completely out the window when it comes to war. Normal time: Killing is bad. In fact it’s pretty much the worst thing you can do. Wartime: Killing is great! Kill as many people as you can! If you’re really good at killing, you get a medal! (Just so long as you kill the right people.) Normal time: Slavery is a blight upon…
AI companies want to bootstrap weakly-superhuman AI to align superintelligent AI. I don’t expect them to succeed. I could give various arguments for why alignment bootstrapping is hard and why AI companies are ignoring the hard parts of the problem; but you don’t need to understand any details to know that it’s a bad plan. When AI companies say they will bootstrap alignment, they are admitting…
I’ve been playing a lot of MTG Arena lately, but I refuse to spend any money on it, which means I can’t craft many rare cards. When I look up meta decklists , they always include a lot of rares and mythic rares. I don’t want to spend all my rare wildcards on one deck! That’s sort of what the Pauper format is for. Pauper decks are only allowed to use common cards, which makes them cheap. But that…