Recent content on jeffcarp
In the course of my work I found myself looking up the specs of various TPUs and GPUs quite frequently.
There wasn’t a good resource that combined the two in a common table with all the relevant stats, so I made one.
It may not be useful to anyone else, but I’ve been using it at least once a day.
 Check it out here and let me know if you find it useful. :)
This is an ongoing list of the best articles I’ve read this year. 
 
 Allow me to get to know you, mistakes and all
 
 TPU Topology Visualizer 
 My time at Amazon: Part one and two 
 1-bit Hokusai’s ”The Great Wave” 
 RTX 5090 + M4 MacBook Air: Can it Game? 
 Difficult, Dated, Frustrating, Prophetic: Teaching Thomas Pynchon 
 Stupid Party Ideas for Your Silly…
I published an account of an interesting* yak shaving
journey I
went on to debug a custom Pallas kernel on TPU. I’ll probably inline the
content here at some point. 
 *Maybe to me and two other people on Earth

 Disclaimer: The views and opinions expressed in this post are my own and do
not necessarily reflect the official policy or position of my employer. 
 
 As someone who’s worked with TPUs for a while, I’ve seen a common pattern in
people coming from the GPU world: they’ll take a look at a TPU chip’s specs,
then look at a comparable GPU, and see…

 I’ve been a die-hard vim user for 12 years. But recently, inspired by posts
from Julia
Evans 
and Felix Knorr ,
I have been trying out Helix –a relatively new
terminal-based editor written in Rust–and I’ve been pleasantly
surprised by the experience. 
 Things I like 
 It’s snappy! Even though some of the command flows have…
Over the years I’ve used a variety of different note taking apps and systems: 
 
 2008: During college I took mainly handwritten notes. I used Apple Notes for casual snippets 
 2012: Mostly kept Markdown notes in a directory 
 2016: Switched from iOS to Android and used Google Keep for notes 
 2020: Moved to Notion 
 2022-present: I’ve been using Apple Notes as my primary…
Last week I attended an AI product summit at the beautiful AGI House on the
peninsula. It was a fun day getting exposed to what’s going on the frontier of
product and AI. Here are my semi-structured notes. 
 
 
 
 The thorniest issue in tool use is identity and authentication
 
 Authentication bugs are much worse with agents because they can move
faster…
This is an ongoing list of the best articles I’ve read this year. 
 
 Tensor Parallelism with jax.pjit - a super intuitive explanation of tensor parallelism 
 The Annotated Transformer 
 Attention From First Principles 
 Ten Rules for Negotiating a Job Offer 
 Note to my present self 
 Video: I’m Politely Begging You to Write Nonfiction 
 21 Facts About…
Here’s how to emulate multiple CPUs when running JAX. This makes it easy to
test multi-TPU/GPU code without actually needing the accelerators. 
 import os
 os . environ[ 'XLA_FLAGS' ] = (
 os . environ . get( 'XLA_FLAGS' , '' ) + 
 ' --xla_force_host_platform_device_count=8' 
 )
 
 import jax
 jax . devices()
 [CpuDevice(id=0),
…
Last year after rejoining Google , I became
a manager for the first time. As a long-time IC, I had some preconceived
beliefs about what the job would be like, many of which were wrong. 
 During this transition a mentor pointed out one chapter from Nine Lies about
Work about
how humans are terrible at objectively rating other’s behavior. The result is
that your entire work…
In 2018 while studying ML and
 cryptography 
around the same time, I realized that many cryptographic algorithms can be
expressed as computation graphs, the same ones supported by major ML
frameworks, which led to a completely frivolous attempt to implement
cryptographic algorithms in
TensorFlow , just to see if it would
work. 
 The world has changed a lot since…
Keeping it short and sweet this year. Here are my highlights: 
 
 Welcomed a baby into the world!!! 
 Recovered enough to run again 
 Ran 138.5 miles 
 Left Waymo and Joined Google 
 Became a manager 
 Was mentioned in TechCrunch 
 Went to Hawaii for our babymoon 
 Went to Taiwan and China to visit in-laws and got to pick tea on a tea farm 
 Read 21 books…
Recently I’ve been trying to learn more about JAX, the next-gen ML framework
from DeepMind. These are the resources I’ve found most helpful so far. 
 
 The JAX docs , which are excellent 
 Patrick Kidger’s Learning JAX as a PyTorch developer 
 JAX AI Stack tutorials 
 The book JAX in Action , along with accompanying Python notebooks 
 Codebases: Gemma , MaxText…
Here’s a list of articles from this year that resonated with me. 
 
 Brighter Than a Cloud 
 Five Principles from Renaissance Technologies 
 Panic! at the Job Market 
 PyTorch is dead. Long live JAX. 
 The Future of Compute: NVIDIA’s Crown is Slipping 
 lanparty.house 

A while back I put together a personal “top 10” list of trail runs I want to
complete in my lifetime. In terms of running goals, I find myself drawn less to
the big races and time goals, and more to the thing that makes me want to get
out the door and run in the first place: running in beautiful places and taking
on technical and logistical challenges. This is my list…
Here’s a quick list of my favorite software engineering books I’ve read over
the years. I plan to keep this list updated as I discover (or am recommended,
hint hint) more. 
 
 Crypto: How the Code Rebels Beat the Government—Saving Privacy in the Digital Age , Levy 
 Cryptography Engineering: Design Principles and Practical Applications , Ferguson, Schneier, Kohno…
I wanted to take a second to reflect on the tone and content of this blog, which overall I’m not super satisfied with. This blog has primarily served as a place where I put mostly software engineering and running content over the years. However, rarely do I compose a post for this blog that I feel is “good enough” to be publishing here. 
 This goes beyond perfectionism — I think seeing amazing…
I’ve observed during my software engineering career the different reactions
senior engineers have when new ideas are brought up. 
 Sometimes when an idea is presented, usually by a junior engineer, a senior
engineer will chime in very quickly to emphatically describe why it won’t work.
I think this is very harmful to team culture for a couple reasons. 
 First, it reinforces the…

 
 
 Joining Google in 2016 vs. 2024. 
 

 This Spring I joined Google for the second time. I’m working on core ML
infrastructure, specifically ML frameworks for Keras, JAX, and Gemini. 
 Not everyone is aware that Waymo is a completely separate sister company to
Google. Going between the companies is an external hire in both directions. 
 Since joining…
I really enjoyed this
book .
Here are some of the biggest takeaways for me. 
 As a manager, your work is meetings. 
That means not protecting your calendar as much. People come before focus
blocks. Corollary: making sure you can perform at a high level in meetings;
meaning exercising in the morning or doing what it takes to make you the
happiest and best person in…
Hi, I’m trying to blog more this year. Maybe not every day, but more
frequently. 
 In a recent post I mentioned I’ve
been working on a fast frontend for a Chinese-English dictionary. In the spirit
of “blog about what you’ve struggled
with” ,
the biggest challenge I’m currently facing is choosing between designing the
site as front-end heavy or back-end…
Hi there, it’s time to reflect on what I was up to this year. 
 Running 
 I started 2023 thinking I had a long-term knee injury. Around Summertime I made a breakthrough in discovering it wasn’t a knee issue but a toe issue. Since then I’ve been keeping the mileage low and trying to build a healthy base. 
 In total I ran 361 miles this year, which is down from usual but I was very…
Recently I’ve been hacking on ChineseDict , a
fast web frontend to look up Chinese words. I started the project originally to
fulfill a specific use case: when practicing chatting with friends or watching
a video, I need search results in milliseconds, not seconds. 
 The website uses the CC-EDICT dictionary to
power the results. I now want to expand each term with…

 
 
 John Cage’s 10 rules for students and teachers
 
 
 Kevin Kelly - 103 Bits of Advice I Wish I Had Known 
 Kevin Kelly - 68 Bits of Unsolicited Advice 
 Dru Riley - 100 Rules To Live By 
 Khe Hy - 43 life lessons at age 43 
 Octaphilosophy: The Eight Elements of Restaurant André 
 Thelonious Monk’s 25 Tips for Musicians 
 Ideopunk - 100 Tips for…
Hi friends, 
 Like last year , here’s a brief overview of
what I got up to in 2022. 
 Running 
 Despite this being my biggest running year in terms of mileage (683mi), I
struggled with injuries pretty much the whole year, likely due to too much
racing and not enough
training . 
 My “goal race” for this year was running the Four Pass Loop, a 28mi loop…
When I have a running injury, it’s not always the right thing to completely stop running and wait for it to heal. 
 Sometimes it’s worse to stop running altogether, because while the injury is healing, all the surrounding muscles are getting weaker. This actually makes injury more likely when I return to running. My current strategy is to keep running as much as possible as long as the pain is…
Inspired by Julia Evans’s post Questions I’m asking in
interviews 
and Chip Huyen’s Questions to ask your
interviewers 
section from her ML Interviews book, here are the questions I ask in
interviews, targeting staff roles in ML infrastructure. 
 Overview 
 My general approach is: 
 
 Have 3-5 “primary” questions that cover the most important things. 
…

 
 
 View from the second peak. 
 

 While on vacation in Hawaii I was super pumped to get in a nice trail run. Here’s how it went. 
 Course 
 
 
 
 
 
 
 
 
 
 Distance 
 5.3 miles 
 
 
 Elevation gain 
 2326 ft 
 
 
 Average grade 
 12%(!) 
 
 
 Duration 
 2:32:43 
…
Happy new year! Here are my 5 favorite books of 2021. 
 Empire of Pain 
 A sprawling, enthralling story of the opiate epidemic told through the lens of
multiple generations of the family that ushered it in. This book reminded me
of Bad Blood , but
with far larger scope. 
 
 In his book Opium: A History, Martin Booth observes that when it comes to
products derived…
I wasn’t initially planning on writing a year in review blog post for 2021, but
I was recently reminded of how nice it is to have a concise public record of
the things that were important to me for each year
(even if nobody other than me reads it). So here it goes! 
 Highlights 
 
 Getting vaccinated! 
 Going back out to restaurants and dining 
 Attending weddings…
Let me state the obvious: recruiters aren’t actually sending most recruiter
emails. They automate it with software. However, the recipients of recruiter
emails must spend real human time either responding to or ignoring these
emails. Unfortunately just ignoring the emails triggers more followup emails
(they usually come in sets of 3). This adds up to a lot of wasted time. 
…
Anki and Spaced Repetition 
 I’ve adopted the Anki app to help learn new
vocabulary and retain various things I learn Currently my decks are: 



 
 
 
 My Anki decks 
 

 Spaced repetition flashcards are great. I would have done much better in school
if I had known about them. 
 Investing and Indexing 
 For most of my 20s I…
2020. What can I say? It was a year I don’t think any of us need to live again. Here’s a brief overview of my year. 
 Highlights 
 In February on Valentine’s day, Elva and I adopted Noona(!), an 8 year old Siberian Husky. She’s the chillest, sweetest dog. Due to lockdown starting 2 weeks later, I’m sure she thinks that spending all day at the house giving her…
I recently went through this process and couldn’t find a guide (though I swear one existed at some point in the past). Here’s how to clone a git repository from Google Source Repositories in the Working Copy iOS or iPadOS app: 
 
 Navigate to https://source.cloud.google.com/ 
 Pick out the repo you want to clone and open the clone dialog (it looks like a “+”…

 Despite this trip report being over 9 months late, I wanted to share it because
I can’t stop thinking about how positive an experience this conference was.
Grace Hopper is the largest women in tech conference in the world, with around
25,000 attendees flying into Orlando, FL from all parts of the world for the
2019 conference. In previous years I had been interested in…

 A bot who can write a long letter with ease, cannot write ill. 
 
 —Jane Austen, Pride and Prejudice 
 这篇文章将引导您逐步学习如何使用Python从头开始编写马尔可夫链( Markov
Chain )，以生成好像一个真实的人写的英语的全新句子。
简·奥斯丁的《傲慢与偏见》( Pride and Prejudice by Jane
Austen )
是我们用来构建马尔可夫链的文字。
 Colab 
上有一篇可运行的笔记本版本。 
 
Read
 the English version of this post here .
 
 Setup 
…

 I think the most effective way to learn a language is to prioritize learning
the day-to-day most frequently used words. Picking words to study in order of
frequency is the optimal way to maximally increase your marginal understanding
of the language for each successive word you learn. 
 To that end, I took a dataset of Weibo (China’s Twitter) posts and ranked…

 
 
 center-384 
 

 
 
 
 The end of the pain cave. Note: the clock time in the picture is for a different race.
 
 
 I PR’d my half marathon at the Napa Half!! Yeahh!!!! 
 
 Old PR: 1:31:37 at the 2019 Kaiser SF Half . 
 New PR: 1:28:59 ( link ). 
 Improvement: 2.9% (-2:38) 
 
 In this post I want to look into the…
This is a “brag doc” to contain all my
accomplishments in a more long-winded and media-rich format than my resume can
handle. 
 Experience 
 Waymo 
 2019 to present — Mountain View, CA 
 
 Gained Python readability. 
 (More here later) 
 
 Google 
 2016 to 2019 – Mountain View, CA 
 
 Built and launched wpt.fyi 
 
…
ColbyBox (2012) 
 I turned an infrequently used LCD screen in a prominent location in our college’s student center into a Twitter live feed that also supported photos and videos. 
 




 
 
 The ColbyBox, an LCD in the student center. 
 


In 2019 I got married—twice! (to the same person!) I changed jobs. I ran my first marathon. 2019 was a huge year in my life. 
 I grew in a lot of ways this year. Wedding planning was a huge undertaking and strengthened our communication and quick decision making skills. Additionally this year I started working with a therapist which has been great—I’m wondering why I never…

 Here are my 4 favorite books from 2019. 
 1. 
 A British couple buys a farmhouse in the South of France and spends the next 12 months exploring the countryside, meeting the locals, renovating the house, and of course, eating and drinking well in [A Year In Provence]( https://www.goodreads.com/book/show/ 40189.A_Year_in_Provence) by Peter Mayle. For a premise that could come off as a…
I did it!! I ran my first marathon! It was painful… but worth it! Despite
still being pretty sore I can tell you I would definitely do it again. Solid
 Type II fun . 
 My final time was 3:59:44 , sneaking in just under 4 hours. You can view the
full splits
 here 
and my run on Strava here . 
 
 Despite consciously trying to keep my pace under control in the…
I’m signed up for the California International Marathon (a.k.a. CIM) in
Sacramento, which is in less than 3 weeks! This will be my first marathon
(the SF Marathon this year was supposed to be, but I gave myself a toe injury
by overtraining ). 
 My goal is simply to finish. I think having any sort of time-based goal would
risk pushing me past the point of injury during the…
Quick life update: I’ve left the Chrome team and joined Waymo (formerly the
Google self-driving car project). I’ll be working on ML infrastructure. 
 It was a fantastic whirlwind 3 years working on infrastructure for Chromium and
helping to–in a very small way–push the open web forward. On the team I
launched wpt.fyi , a resource to help align the APIs of…

 I hosted an intern for the first time this summer. It was my first time being
somebody’s manager and it became a huge learning experience for me as well as a
really fun time. My intern worked on adding many features to velocity-tracking
charts ,
 rewriting our ML models in TensorFlow
2.0 ,
and a few other projects. 
 Here are the biggest areas where I…

 Note: This post had a good run, but it is now very out of date.
 Check out newer content around tokenizers 
 for a more up-to-date view.
 
 Machine learning on human languages is a super exciting space right now.
Applications are exploding—just think of how many natural language ML models
it takes to run a smart assistant, from transforming spoken audio to…
One year ago I conducted my first software engineering interview at Google. In
that first interview I gave, I guarantee you I was more nervous than the
candidate I was interviewing. Those first few interviews were particularly
nerve-wracking. A lot was on the line—I didn’t want to screw up this person’s
career by being a bad interviewer! 
 Since then…
tl;dr: 
 I signed up for the SF Marathon (this would have been my first marathon), then
overtrained, got injured, and am currently recovering. I’m probably going to
defer my registration to 2020 and become a cheering squad this year. 
 (╯°□°)╯︵ ┻━┻ 
 This is a cycle I’ve been through over and over again. I literally
 wrote about
this in 2015 . Being injured…
If you’re a software engineer, you’re likely familiar with unsolicited emails
from recruiters. Most are probably template emails. Some of them are funny,
some are thoughtful, and some of them ask you to move 3000 miles, take a 50%
pay cut, and code in a language you don’t know. 
 Impact 
 Recruiter emails have a measurable impact on productivity. If I were…