Blog posts by Thomas Broadley.
I. Like most people, my teens and twenties have been confusing and not always the most fun. I’ve struggled to make friends. In high school and university, I didn’t have as many romantic relationships as I wanted. When I was 24, I met a beautiful, wonderful woman who became my wife, but I still feel like I have a lot of room to be a better husband. I lucked into a relatively stable and interesting…
Last June or July, I switched from working primarily on METR's AI evaluations infrastructure to working on a particular research project. Last week, my team published a blog post explaining our research progress so far! You can read it here: https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations/ Our goal is to evaluate the monitorability of AI systems. METR wants to be able to…
Even in Berkeley, we get less than ten hours of sunlight a day in January. I feel bleh when the sun sets at 5pm. So, last year, I built a lumenator! It's a string of 25 13-watt (100-watt-equivalent), 1,500-lumen lightbulbs hung from the top of my apartment walls using Command hooks. I put paper lanterns on the lights. They're aesthetic and, when I look directly at the lights by accident, they…
About a month ago, METR published a paper called "RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts". RE-Bench evaluates human experts and AI agents on machine learning research engineering tasks. Research engineers at Anthropic, Google DeepMind, and OpenAI perform these kinds of tasks in the course of their jobs. The benchmark's goal is to measure…
My wife and I just moved to Berkeley, California. I'm working in-person at METR while my wife creates music and online content. We'll be here for a year. If you live in or visit the Bay Area, please let me know! You can contact me through the email on this website's main page .
METR just open-sourced Vivaria , our platform for writing AI evals and running agents on them. Over the last ten months at METR, Vivaria is the software project to which I've contributed the most. I'm surprised how happy I am that it's public. I think it's because I'll find it much easier to talk about my work, now that I'm mostly working in public! If you have questions about Vivaria or are…
At METR, I'm trying to reduce existential risk from artificial intelligence by helping to measure the autonomous capabilities of language model agents. For the past two months, I've helped METR develop a standard for tasks that evaluate these agents: the METR Task Standard . The standard formalizes METR's internal task format, now available for other people and organizations to use. Writing good…
In the last couple of weeks, I've given two talks on the subject of dangerous capabilities evaluations for AI, one to Toronto AI Safety and the other to the Wisconsin AI Safety Initiative . In both talks, I discussed dangerous capabilities evaluations: tests for AI systems that check whether they can make it easier for individuals or small groups to develop biological, chemical, or nuclear…
OpenSSH (the book) says this about SSH multiplexing: Multiplexing is the ability to send more than one signal over a single line or connection. In OpenSSH, multiplexing can re-use an existing outgoing TCP connection for multiple concurrent SSH sessions to a remote SSH server, avoiding the overhead of creating a new TCP connection and reauthenticating each time. For example, maybe you want to use…
A little over two months ago, I left my job at Faire to figure out how I could help reduce existential risk from AI . Today, I'm happy to announce that I've accepted a position as a Member of Technical Staff at ARC Evals , a project of the Alignment Research Center . According to its website, the project's goal is to "assess[] whether cutting-edge AI systems could pose catastrophic risks to…
In preparation for moving to the US, I purchased traveller's health insurance for my wife and myself. This insurance doesn't cover preventative medicine, like yearly checkups and vaccines. I was prepared to pay out-of-pocket for the updated COVID vaccine until I heard about the CDC's Bridge Access Program . The program's website states, "Adults 18 years and older without health insurance and…
Cross-posted to LessWrong . I reproduced results from ARC Evals ' recent report, Evaluating Language-Model Agents on Realistic Autonomous Tasks . For the report, ARC Evals built a set of language model agents, combining a language model like GPT-4 with scaffolding software that lets the language model execute shell commands and interact with a browser. Then, it asked the agents to complete a set…
In just over a week, I’m quitting my job as a software developer at Faire. Over my four-and-a-half years at Faire, I’ve seen the company undergo an order of magnitude of growth in both employees and customers. I’ve learned a lot from many excellent mentors and managers. It’s a bittersweet departure. Why am I leaving? For one thing, in less than a month, my wife and I are moving to Los Angeles so…
Last week, I created a chatbot that answers questions about AI safety. You can try it out here: https://ai-safety-conversational-agent.thomasbroadley.com/ The source code is on GitHub . I made the chatbot because of a bounty on LessWrong, a website about rationality, AI safety, and existential risk. The chatbot answers easy questions about AI safety, such as "what is AI safety?". It can also…
I started learning Vietnamese on Duolingo three months ago. My goals are simple: to speak Vietnamese with my girlfriend's family, to communicate with locals on our future trips to Vietnam, and to teach Vietnamese to any children we might have. Duolingo offers exercises to learn vocabulary, translate sentences between English and Vietnamese, and transcribe spoken Vietnamese. I've already learned…
A few days ago, a coworker and I played around with ChatGPT for 10 minutes. We discovered that it isn't half-bad at helping you practice for systems design interviews. Here's part of a conversation I just had that demonstrates this: Thomas: Can you help me practice for a systems design interview? You pretend to be the interviewer, I'll be the candidate. The interview question is, "design a…
I’ve written a lot of Kotlin code over the past three years, but I didn’t get Kotlin’s in and out keywords… until yesterday. Here I’ll record my understanding so hopefully I don’t forget how they work. I’m basing this on Kotlin’s documentation on generics. in and out on classes in and out can appear in two different places and have slightly different meanings depending on the context. First, they…
I’m working through Hammertime , a 30-day program of instrumental rationality exercises. Hammertime introduced me to a technique called Focusing via a LessWrong post called “Focusing, for skeptics” . Focusing involves: Thinking about a problem Paying attention to how your body feels. This is a felt sense Coming up with a handle for that felt sense: a word, image, or concept Comparing the handle to…
I subscribe to about 100 RSS feeds and one email newsletter: Money Stuff. I wanted a way to receive Money Stuff in my RSS reader instead of in my inbox. Unfortunately, Bloomberg doesn't seem to offer an RSS feed for Money Stuff anymore. I found a website called Kill the Newsletter that converts email newsletters into RSS feeds. For each newsletter you want to subscribe to, Kill the Newsletter…
In "Jumping in the deep end with Workman" , I concluded: I intended to title this post "Mastering Workman", but I don't think I've reached mastery yet. Hopefully next time I report back I'll feel that way! I'm happy to say that I feel that way now! I just typed at 90 wpm in a typing test. Based on the previous post, that's just below or the same as my QWERTY typing speed in 2020. I'm close to…