How to win a best paper award
An opinionated perspective on how to do important research that makes a difference (and sometimes win awards).
Nicholas Carlini
Nicholas writes things.
Latest posts
Are large language models worth it?
Are the harms that LLMs have caused, and will soon cause, worth the benefits they may bring? This article (a written version of a keynote talk I gave at CoLM) tries to explore this question.
Gate-level emulation of an Intel 4004 in 4004 bytes of C
A feature-complete gate-level microcoded Intel 4004 in 4004 bytes of C, capable of emulating the original Busicom calculator ROM for which the chip was originally designed.
miniHDL: A Python Hardware Description Language DSL
A small hardware description language implemented as a DSL on Python, with a small 170 LoC 32-bit RISC CPU.
Machines of Ruthless Efficiency
Future LLMs have the potential to cause significant harm due to their ruthless effiency. I'm worried this will happen, and discuss the ways in which it might.
My Thoughts on the Future of "AI"
I have very wide error bars on the potential future of large language models, and I think you should too. It's possible LLMs basically lead to AGI, and it's also possible they platteau.
What my privacy papers (don't) have to say about copyright and generative AI
My work on privacy-preserving machine learning is often cited by lawyers arguing for or against how generative AI models violate copyright. This maybe isn't the right work to be citing.
Career Update: Google DeepMind -> Anthropic
I have decided to leave Google, and will be joining Anthropic to continue my work on machine learning security
AI forecasting retrospective: you're (probably) over-confident
A one-year review of people's predictions on an AI-forecasting survey I made last year. Most people were over-confident in their predictions.
A 2-ply minimax chess engine in 84,688 regular expressions
I wrote a (list of) regular expressions that will play a (not very good) chess game by running a 2-ply minimax search.
Letting Language Models Write my Website
I let a language model write my bio. It went about as well as you might expect.
You should forecast the future of AI
You should forecast the future of AI in this quiz, so that you can see just how right or wrong you are.
How I Use "AI"
I don't think that AI models (by which I mean: large language models) are over-hyped. In this post I will list 50 ways I've used them.
Why I attack
Yesterday I was forwarded a bunch of messages that Prof. Ben Zhao (a computer science professor [a] A full professor with tenure, so I feel entirely within my rights to call him out here. at the University of Chicago) wrote about me on a public Discord server with 15,000 members, including this gem:
(yet another) Broken Adversarial Example Defense at IEEE S&P 2024
IEEE SP 2024 (one of the top computer security conferences) has, again, accepted an adversarial example defense paper that is broken with simple attacks. It contains claims that are mathematically impossible, does not follow recommended guidance on evaluating adversarial robustness, and its own figures present all the necessary evidence that the evaluation was conducted incorrectly.
My benchmark for large language models
A benchmark of ~100 tests for language models, collected from actual questions I've asked of language models in the last year.
My research idea logfile, 2016-2019
How do I pick what research problems I want to solve? I get asked this question often, most recently in December at NeurIPS, and so on my flight back I decided to describe the only piece of my incredibly rudimentary system that's at all a process. I maintain a single file called ideas.txt, where I just append a new entry every time I think of something that might be an interesting topic for a…
Reading Data off an Apple ProFile Hard Drive with an Arduino
So let's suppose you had a 1980s Apple ProFile Hard Drive, and you wanted to recover the data.
Playing chess with large language models
Building a chess bot that queries GPT-3.5-turbo-instruct to play chess at the level of a skilled human player.
Little Bobby |endoftext|
TODO
A ChatGPT clone, in 3000 bytes of C, backed by GPT-2
This program is a dependency-free implementation of GPT-2, including
Reflecting on Towards Evaluating the Robustness of Neural Networks
I recently got back from attending USENIX Security 2022, and someone pointed out to me that it's been five years since I wrote Towards Evaluating the Robustness of Neural Networks (with my at-the-time advisor) and they asked if I had any thoughts on this paper. I didn't respond with that great an answer, but thought it was an interesting question, so I figured I'd write one down here instead. (In…
Rapid Iteration in Machine Learning Research
A brief discussion about a tool I use to make rapid iteration in ML research possible.
A Case of Plagarism in Machine Learning Research
A recent paper ('A Roadmap for Big Model') has copied a bunch of text from over a dozen prior papers. This is bad.
Multiplexing Circuits on the Game of Life - Part 5
Abstract: Improving digital logic gates on Conway's game of life by allowing 8-bit logic gates instead of boolean logic gates.
Research Paper Release Checklist
This page contains a few checklists that help prevent embarrassing issues when releasing research papers online (e.g., via arXiv or a conference publication).
A Simple CPU on the Game of Life - Part 4
Abstract: An implementation of a minimal CPU on Conway's the Game of Life (an 'unlimited register machine'), and runs at ~10Hz.
Improved Logic Gates on Conway's Game of Life - Part 3
Abstract: This post describes improvemnets made to my prior digital logic gate constructions (e.g., AND/OR/NOT) built on top of Conway's Game of Life, resulting in 100x faster simulations.
Yet Another Space Game (In 13kb of JavaScript)
This year I entered in JS13K 2020, a game jam for JavaScript games in under 13KB (total size). I wrote a 3rd-person space shooter game, building on top of game engine I built last year for a doom clone.
InstaHide Disappointingly Wins Bell Labs Prize, 2nd Place
InstaHide (a recent method that claims to give a way to train neural networks while preserving training data privacy) was just awarded the 2nd place Bell Labs Prize (an award for finding solutions to some of the greatest challenges facing the information and telecommunications industry.). This is a grave error.
Yet Another MOBA (In 13kb of JavaScript)
For the third year in a row, I participated in JS13k 2021, where you're tasked with making a game in 13kB of JavaScript. Each year I enter participate I try to learn something new I didn't know how to do before. This year's motivation: I wanted to make a multiplayer game with some nontrivial networking aspects. So below you'll find a description of my experience building a MOBA in (barely) under…
Realtime Screen Recording of Breaking a Defense to Adversarial Examples
I recently broke a defense to be published at CCS 2020, and this time I recorded my screen the entire time---all two hours of it. Typically when I break defenses, I'll write a short paper, stick it on arXiv, and then move on. Pedagogically, this isn't very useful. [a] (Don't you worry, I did that again this time, too.) So for this defense I thought I'd try something different.
An Introduction to Circuit Design on Conway's Game of Life - Part 2
Abstract: Using AND/OR/NOT gates built on top of Conway's Game of Life, this post walks through how to construct a actual circuits, for example a 7-segment display.
Digital Logic Gates on Conway's Game of Life - Part 1
Abstract: This post walks through how to construct digital logic gates (AND/OR/NOT) on top of Conway's Game of Life, demonstrating its Turing completeness.
Are adversarial example defenses improving?
Abstract: We (again) broke a large collection of published defenses to adversarial examples. Here's how and why.
Yet Another Doom Clone (In 13kb of JavaScript)
This year I entered in JS13K 2019, which asks people to develop games in under 13K of JavaScript. I entered a Doom Clone called ... Yet Another Doom Clone.
3D Shadow Mapping Renderer in JavaScript
Late last year I decided it would be fun to build a 3D renderer in JavaScript. Recently it got into some sort of finished state and decided to put it here. This isn't so much of a tutorial on how to get there, but rather more of a here's a fun thing to do with nice pictures. But it was interesting to do. So here's that.
A Complete List of All (arXiv) Adversarial Example Papers
Abstract: A continuously-updating list of all 1000+ papers posted to arXiv about adversarial examples.
Adversarial Machine Learning Reading List
Abstract: This reading list provides an introduction to the field of adversarial examples for machine learning models.
Recommendations for Evaluating Adversarial Example Defenses
Abstract: This document contains a collection of advice for performing adversarial example defense evaluations.