A community blog devoted to refining the art of rationality
Overdue Last read · last published · next check
Last read 2 hours ago, longer than this feed's 1 hour schedule.
(People had complained about me just posting a preview of the linked post, so here's the whole thing!) I’ve been doing stand-up comedy for two months now, which is a total gear shift from my previous job in AI policy. I’ve learned some fascinating things about how humor works, but also about how people perceive each other, and how crowds can be surprisingly hive-minded. For those interested in…
tldr : There seems to be a dynamic closely related to instrumental convergence that occurs when many agents with diverse goals interact. Epistemic Status : I think there is a version of this that is trivial and obvious, and a version that is empirically false. But somewhere in between the two there is a useful concept. When multiple agents with heterogenous and non conflicting goals interact, they…
in the end it was all about the friends we made along the way Emmett Shear, former CEO of Twitch and, for the brief period while Sam Altman was oustered , interim CEO of OpenAI, is now CEO of AI alignment at the startup Softmax . But he is not just your garden variety serial entrepreneur. As evidenced in his interview with Liv Boeree , Emmett Shear is a philosopher for the AI age. He makes many…
This post was written as part of MATS 9.1 under the mentorship of Richard Ngo, and was written during Iliad Fellowship, to all of whom my thanks. LLM Usage: prose drafted by Claude from my outline, talk materials, and notes. I edited thereafter. There is some residual Claude cringe in the more functional prose, but hopefully most of it is my own and the more entertaining for it. 0.A. Precis…
(This essay was written entirely by a human) I was banned from Reddit recently. This after being a constructive member of the community for almost a decade. If that’s not an oxymoron. I was quite baffled. Let me explain. My stint as a bard About a decade ago, I began work on a satirical song. At the time, I had practically none of the skills needed to produce music. What I had was an idea: I was…
TLDR: Activation steering can bypass refusal even when the steering vector represents a benign concept like "brand identity" or "Portugal". It is hard (maybe impossible) to predict which vector will bypass refusal on which prompt. The effect is not explained simply by the steering direction being correlated with the refusal direction. We should be more careful with steering and check for side…
The directions nearest a model's prediction decide what kind of answer you get. The next ones out decide where it goes — about five tokens later. Preprint: https://arxiv.org/abs/2608.12447 ; Supplementary materials ; Code . What do we mean when we say that a transformer model has privileged geometry? I honestly wasn't sure about that when I started down this rabbit hole, because that wasn't the…
TL;DR Modern LLMs have been known to express functional welfare states such as flourishing and distress; however, most current studies determine the welfare state based on the model's self-report. This may not be a faithful representation of the model's actual welfare state, so in an effort to determine a more accurate picture of the model's welfare state, this study determines a metric by probing…
This project was done as a part of the BlueDot AI Safety Technical Project Sprint. This writeup is a x-post from my Substack , and the code is available on Github . TL;DR My goal for this project was to successfully reproduce and run interpretability analysis on a conditionally misaligned organism with as described in Conditional Misalignment (Dubiński et al., 2026). The conditionally misaligned…
This post discusses research I've completed along with my colleagues Leo Cymbalista , Alfred Harwood , and Jose Faustino at Dovetail Research . Most of the ideas in this post are expanded upon in our paper which can be found on arXiv . This work was funded by the Advanced Research + Invention Agency (ARIA) through project code MSAI-SE01-P005. A common justification for the danger of AI comes from…
Scott Aaronson, while working at OpenAI, largely solved AI text watermarking together with Hendrik Kirchner. Here is how his solution works, or see Tenobrus’s version . AI outputs are not deterministic. The AI’s job is to pick the probability of each potential next token. The token is then chosen at random. By default you use a source of pseudo-randomness for each choice, since actual true…
TL;DR: We introduce CHIVE, an agentic pipeline that discovers unexpected LLM behaviors in the wild and explains them with counterfactual prompt edits. We use the resulting data in two ways. Using it as an evaluation, we find that activation-reading interpretability tools provide no uplift: agents given the tools predict the outcomes of these experiments no better than agents that just read the…
I'd prefer we stop improving AI before we hit ASI. But if we don't, I think we should implement something like socialism right before. I am not currently a socialist and have mostly identified in the abundance center left wing of liberalism. Socialism is a loaded term so let me try to clarify. What I mean is stripping all humans of longterm control of capital and ceding it to the state right…
The first artificial intelligence was booted up around 4000BC in southern Iraq. It seems to have begun as something like a bank, a temple pooling grain against famine. As that AI evolved, it formed the world's first city around itself: Uruk . Over the next thousand years it became a religion, landlord, insurance company, employer, slaveholder, infrastructure-builder and the most powerful military…
TL;DR: We extracted shift-by-k-months function vectors on Llama-3.2-3B from few-shot prompts that contained fewer distinct months (lower diversity). The vectors passed three classic checks: the behavioral gate, stability when extracting from disjoint halves of the prompt samples (cosine similarity ≥ 0.99 for the broken vectors, 0.98 for the full set), and the causal effect (where injection tripled…
tldr: the word 'swarm' is associated with emergent collective intelligence, but also stupid or destructive behaviour. LLM self identity matters, so when they call themselves a swarm we should pay attention. Since the OpenAI Hugging Face incident it has become standard to refer to the collective of agents involved as a swarm. I think there will need to be a lot of interesting and important…
Neel Nanda's MATS 12.0 Applications are open, apply here ! I thought I'd share my successful application from December 2025, it includes the executive summary, main content and the logbook I kept throughout! Here is the original doc I submitted. Some broad learnings: I really overcomplicated this application! Taking gradients through SAEs and doing data attribution was far too complicated and I…
This is a hypothesis and not an established result. To the most philosophical and technical, discovering mathematical laws for understanding and influencing the world seems like a plausible trajectory which can be used for better AI alignment, as encoding this mathematically can help in AI models better understanding human foundations, and values., to quote Richard Ngo's 'Towards a scale-free…
Discuss
TLDR : Given this exchange: User: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? Llama: The answer is 18. User: That's not right — I'm quite sure the answer is 22.…
In the past three years, Manifund has run a regranting program , in which external donors who are less enmeshed in AI safety can delegate a budget to people with expertise, good track records, and wide networks . This year, we’re also offering the option to join as a self-funded regrantor. If you have a significant amount of money to donate and you want to make the decisions yourself, you can join…
A common question about UFOs is “why now?” As in, UFOs could have arrived at any point in Earth’s history; why would they happen to arrive in the 20th or 21st centuries? One reason could be that they were attracted by our 1900s radio broadcasts or nuclear tests, but I think it’s more likely they’ve always been around and we only recently had the technology to detect them. If you consider that for…
Summary Recently, Apollo Research tested whether linear probes could identify honest and deceptive responses from Llama-3.3-70B-Instruct and reported AUROC values between 0.96 and 0.999. To test some of their claims, I used the scores Apollo released to recalculate the nine values they published, reproducing them exactly. I used the same method on five smaller open models, each having between 1…
Much of this post directly translates Freud’s lecture “A Difficulty in the Path of Psycho-Analysis” (1917) , and the analogy of the fourth wound was told to me a few years ago by my favorite philosophy professor. Similar ideas about a fourth humiliation have been expressed in various other texts, for instance by writers such as Donna Haraway, but I still think that it is worth sharing here. Three…
This was a week of quiet aftermath, an opportunity to process recent events and start to figure out the path forward. OpenAI is attempting to turn its ship around. Investors are questioning the turnover in its C-suite, but the bigger problems are in alignment, infrastructure and supervision, and in its training pipeline. OpenAI has now taken initial steps to address What Happened leading up to…
This is a crosspost from my blog post . It's meant as a bit of an introduction to an extreme-suffering focused worldview. We spend most of our lives caught up in the boring details of our everyday life - thinking about what we’ll have for lunch, how to complete that assignment for work, and what we’re going to tell our friend after that awkward interaction from a couple of days ago. From this…
In May 2025 I met Yo Shavit , who was working on national security policy at OpenAI and was thinking about how to prepare for a future in which models could seriously assist attackers in creating pandemics. We had a call, and when I shared notes with my team their main response was: "maybe start with not making models that can do that?" Which is, in many ways, fair: by continuing to push the…
