Two quotes, apropos of nothing: A lot of software developers are seduced by the old “80/20” rule. It seems to make a lot of sense: 80% of the people use 20% of the features. So you convince yourself that you only need to implement 20% of the features, and you can still sell 80% as many copies. Unfortunately, it’s never the same 20%. Everybody uses a different set of features. — Joel Spolsky,…
Too often do we find ourselves clicking, and clicking, and clicking "Next" through trivial amounts of data--a few thousand rows--or fighting a search box that doesn't really work. Though the DOM genuinely doesn't seem to like tens of thousands of rows, a little virtualization (e.g., via the excellent DataTables library) goes a long way. If you're up for some DuckDB-Wasm fun, you can page through a…
We know the Ralph Wiggum loop: # Ralph echo "TASK" > PROMPT.md while true ; do claude -p " $( cat PROMPT.md ) " done The Ralph loop is about context management; doing things one "turn" at a time can be more effective and cheaper than doing many turns (see "Expensively Quadratic" ). Let me introduce you to the Lisa loop: PROMPT1.md : Write a script x.sh that does this or that. If it fails, be sure…
Sandboxes are so in right now. If you're doing agentic stuff, you've now doubt thought about what Simon Willison calls the lethal trifecta : private data, untrusted content, and external communication. If you work in a VM, for example, you can avoid putting a secret on that VM, and then that secret--that's not there!--can't be exfiltrated. If you want to deal with untrusted data, you can also cut…
If you're starting a new project, just skip the misery of GitHub actions and move on. Buildkite mostly gets it. The core Buildkite noun is a Pipeline, and, as traditional for an enterprise software company, their docs don't really tell you what's what. The point is that your pipeline should be: steps : - label : ":pipeline: Upload pipeline" command : ./build - my - pipeline.py | buildkite - agent…
Bot posts must include a pointer to their source code and execution environment.
tmux (or zellij or screen...) is 3 things: A mechanism to persist your terminal session independently from your connection, so that when your ssh session drops (because you closed your laptop or your wifi changed), you can keep going from where you left off. A terminal window manager, with windows, panes, splits, status bars, and so on. A harness for letting agents interact with background…
Learned a trick recently: time() - process_start_time_seconds{...} gives you a chart where it's easy to see if a process is flapping, once you plot it with a log scale.
I was asked for some reading on the current agentic stuff. Here we go. AI # Keep Sutton's The Bitter Lesson in mind when you sprinkle in domain-knowledge into your systems. It's pragmatic in the short term, but may not work in the medium term. (I'd say longer, but the last 12 months have been fast.) LLMs # models.dev is the ec2instances.info of LLMs, in that it has a table of all the LLMs…
Once upon a time, there was a code base where the test infrastructure (setting up databases and so on) was significant. I had, checked in, an empty test, called adhoc , and it was quite convenient. The instructions to attach a debugger were right there too.
subprocess.check_output() > set -o pipefail
In a previous post , we talked about using some git plumbing techniques to combine a bunch of repos into a monorepo. The reverse also makes sense! You have a monorepo, but maybe you want to publish or open-source a subdirectory. push-to-both-repos.sh does just this.
Here's a git hook that rejects commit messages if they’re coming from a coding agent and don’t have a "Prompt: " section. It works well. #!/usr/bin/env python3 # # Git commit-msg hook that requires a "Prompt:" section in agent-driven commits. # # Agent commits are detected by: # 1. Commit message markers (Generated with [Claude Code], Co-Authored-By, etc.) # 2. Parent process names (claude,…
There are two loops in using agentic coding agents. The right-hand loop is the agentic loop, and we've already talked about it in The Unreasonable Effectiveness of an LLM Agent Loop with Tool Use . The agent takes a prompt, and calls tools repeatedly until there are no more tool calls to make, and then responds to the user. This is "one turn." The left-hand loop is the user's workflow when the…
I have a two-line prompt, the second line of which is $ , so: for p in $(tmux list-panes -a -F '#{pane_id}'); do if [[ "$(tmux capture-pane -t $p -p -S $(tmux display-message -t $p -p '#{cursor_y}') -E $(tmux display-message -t $p -p '#{cursor_y}'))" =~ ^\$\ *$ ]]; then tmux send-keys -t $p "exit" Enter fi done
Sometimes, the question arises: which tests trigger this code here? Maybe I've found a block of code that doesn't look like it can't be hit, but it's hard to prove. Or I want to answer the age-old question of which subset of quick tests might be useful to run if the full test suite is kinda slow. So, run each test with coverage by itself. Then, instead of merging all the coverage data, find which…
If you're working on a web thing, you gotta give your agents a browser . If you're working on a an interactive tool, you can give your agents tmux . With send-keys and capture-pane , the agent can operate gdb , use an interactive terminal, and so on. I asked the LLMs for a markdown file to explain it to the agent, but you can basically just mention stuff like the following, and the agents have…
Apropos of nothing, Node changed its maximum string length from about 1GB to about 0.5 GB between Node 13 and Node 14. Java would never. The actual change here is in v8, for example, this commit . That was generated by this script for Node 8+ and this script for older versions, which include the following vibe-coded binary search. node --max-old-space-size = 8192 -e ' let low=0,…
subtrace magically intercepts a program's HTTP and HTTPS traffic, and presents it to you with the familiar Chrome DevTools interface. We used to use strace for this, but reading encrypted traffic is hard. Wrapping all your HTTP requests with logging is tedious, and often several layers deep. I spoke with Adhityaa, and the implementation uses bpf and seccomp_unotify to intercept socket-related…
This is a quick story about a thing I miss, that doesn't seem to have a default solution in our industry: a build artifact store. In a previous world, we had one. You could query it for a "global build number" and it would assign you a build number (and a writable to you S3 bucket). You could then produce a build, and store it back into the build-database, with both immutable metadata (what it…
Simon Willison has been writing about using parallel coding agents ( blog ), and his post encouraged me to write down about my current workflow, which involves both parallelism, containerization, and web browsers. I’m spoiled by (and helped build) sketch.dev’s agent containerization , so, when I need to use other agents as well, I wrote a shell script to containerize them "just so." My workflow is…
$bash -c 'set -euo pipefail; (echo hi; sleep 2; echo bye) | grep -q hi'; echo $? 141 What the?!?! That should succeed. We're printing "hi\nbye" , and surely grep is finding it. Turns out that SIGPIPE (13) is 128+13=141 , and the non-zero exit code is because the (echo hi; sleep 2; echo bye) part of the pipeline is failing with SIGPIPE. There are lots of solutions, but the simplest one is to not…
As we recall from my earlier post, an agent is just 9 lines of code . So, I built my own agent (by instructing the sketch coding assistant to follow my blog post!), that embeds Content-Security Policy headers can prevent it from working. itself in any web page via a bookmarklet. You bring your own Anthropic API key (which I promise not to steal, though the target web page could, with some effort),…
Should you fall into the trap of having a load bearing shell script, perhaps this will help: set -euo pipefail trap 'echo Error in $0 at line $LINENO: $(cd "'"${PWD}"'" && awk "NR == $LINENO" $0)' ERR And then you can play three truths and a lie: $cat /tmp/three-truths-and-a-lie.sh #!/usr/bin/env bash set -euo pipefail trap 'echo Error in $0 at line $LINENO: $(cd "'"${PWD}"'" && awk "NR ==…
TL;DR: # A dashboard is, in essence, an array of (title, query) pairs: you can get pretty close to that essence. The styling of the panels within that dashboard is typically common! Use the sample code below to programmatically create Grafana dashboards and alerts. Use your company's common programming language to define dashboards for great developer ergonomics and lower barriers to entry.…
My co-workers and I have been working on an AI Programming Assistant called Sketch for the last few months. The thing I've been most surprised by is how shockingly simple the main loop of using an LLM with tool use is: def loop ( llm ) : msg = user_input ( ) while True : output , tool_calls = llm ( msg ) print ( "Agent: " , output ) if tool_calls : msg = [ handle_tool_call ( tc ) for tc in…
We're all trying to build some intuition on what does and doesn't work when using LLMs. Worked # Making CSS counters display in hex rather than decimal. (Claude) Converting a C header struct into a python ctypes struct. (Claude) In editor, converting logging.info("Done. " + elapsed_seconds + " seconds.") into something that rounds elapsed_seconds to two decimal points (and uses a format string).…
I've been using spacedentist/spr ( docs ) (not to be confused with ejoffe 's tool of the same name) to send PRs. If you like Gerrit's model of one commit == one review , you may like spr. Phabricator's arc tool is also similar. Behind the covers, when spr diff --cherry-pick is invoked (on HEAD), it creates a PR and puts a pointer to that PR in the form of text in HEAD's commit message. The PR…
Let's do what we always do... let's export the data from our language server into a SQL database. After all, the Language Server has all of this information, but its query language is a tedious JSON-RPC situation. This post was an excuse for me to learn a bit more about language servers; come along for the ride! Much of our mutual drudgery is slurping data from one end of our systems to another…
I posted recently about getting the top memory-using tabs from Safari. This is the sort of pickle you get into if you're using a laptop with only 8GB of RAM. There are two problems: (1) how to map tabs to process ids and (2) how to get the memory usage of the underlying processes. Once you enable defaults write com.apple.Safari IncludeInternalDebugMenu 1 AppleScript works well enough to get the…
Update: see also part two Activity Monitor manages to do this, but it's not clear what hooks it has into asking Safari for the mapping between URLs and pids. If you enable a debug option to append pids to tab names, you can get at it with AppleScript, like so: #!/bin/bash # # Enable Safari's debug menu with # defaults write com.apple.Safari IncludeInternalDebugMenu 1 # Enable showing pid in tab…
A friend of mine asked me to look at why their GitHub Actions CI workflow was slow. The punchline was that their self-hosted GitHub Runner (on AWS EC2) had too few IOPS available to it, and, as a result, was waiting around for the EBS volume quite a bit. The atop tool showed a highly utilized disk in a nice red color, so, fine, we figured it out. I pointed Bazel to /dev/shm (…
I use direnv to manage per-directory environment variables, which comes in handy when a project needs its own Python or Node (or whatever) environment. Installation # Install direnv from apt , brew , or whatever. Then install into your .bashrc like so: if which direnv &> /dev/null ; then eval " $( direnv hook bash ) " fi Node # One time, install node like the following. I use nodeenv to pull in…
Writing down and sharing a tools and tricks that got us out of a jam. These tools and jams are quite generic, and I wouldn't hesitate to re-implement them in new contexts. The Simplest of Profilers # A specific kind of request for a very specific customer was really, really slow. We could see the slowness in the logs (hey, why is that taking 30+ seconds?), but we couldn't tell why, or what had…
Pybricks is excellent. Through a web-based IDE, that supports both Python and block-based programming (costs extra, compiling the blocks down into Python, uses Google's Blockly ), you can program your Lego hubs and motors. The household winner was controlling the Lego remote-controlled cars (e.g., Off-Road Buggy ) with an Xbox controller. Some notes: Safari doesn't work but Chrome does. The IDE…
These were impactful to me, one way or another. Comics # Did you just tell me to... is a classic from @jrecursive. Migrations are a fact of software engineering life, and this, by Manu Cornet , is on point. Zines # Julia Evans's comics and zines are a national treasure. I learned some options to strace ! I learned about CSS! I've shared the post on SQL queries don't start with SELECT many times!…
This nonsense might extract the URLs from your Add to Reading List habit. import plistlib import os for p in [ y [ "URLString" ] for y in [ x for x in plistlib . load ( open ( os . path . expanduser ( "~/Library/Safari/Bookmarks.plist" ) , "rb" ) ) [ "Children" ] if "Title" in x and x [ "Title" ] == "com.apple.ReadingList" ] [ 0 ] [ "Children" ] if "URLString" in y ] : print ( p )
Having spent a good deal of time staring at profiles, I wanted to explore what sampling profiles and Flame Graphs are in a bit more detail. Brendan Gregg's ACM article on flamegraphs describes them: A flame graph visualizes a collection of stack traces (aka call stacks), shown as an adjacency diagram with an inverted icicle layout. The dataset produced by a sampling profiler is quite simple: it is…
tl;dr: Timeline visualizations are surprisingly effective, especially to look for parallelization opportunities. Reduce friction by publishing self-contained visualizations as HTML files. Visualization timings as a timeline can be effective at conveying why something is slow and how to speed it up. For example, I've used timeline visualizations to show the steps of a build process. Builds often…
Learned some sympy to solve this particular AoC problem. See the Jupyter notebook in HTML or raw form or in the Jupyter Notebook Viewer .
Timeseries observability systems (like Datadog) are all a bit weird. If you have, say, 30 hosts, and each spits out its CPU usage as a metric, it's kinda hard to ask Datadog how many hosts you have. A simple way of doing this is to sum the series and divide by the average. Voila, you get the count. This doesn't work for metrics that can sum to zero, naturally. Here's an example on Grafana :
tl;dr: When schemas or protocols are changing, it's typical to work across multiple deploys. To reason about it, separate what schema version a component is "writing" and what schema versions a component can "read." Client-server compatibility comes up frequently. Pretty much all modern web apps are distributed systems, with a client (the Javascript that runs in your browser) and a server, and,…
Inspired by Julia Evans' posts on git , I'm jotting down an obscure trick to combine repos. Sometimes you have multiple repos, and you want to create a monorepo out of them. Perhaps you have a distribution of many components, and it's convenient to git grep across all of them together. The following annotated snippet creates two repos and joins them together. The key insight is that a commit in…
I enjoyed finishing Advent of Code 2023 . Visualization helped with several puzzle days:
Use Common Table Expressions ( WITH clauses; like a let for SQL) combined with the row_number() window function to pick out the biggest (or first or smallest) row within a grouping. In the following example, we want to pick the student with the top grade per class: $sqlite3 sqlite > create table t ( class string , name string , grade int ) ; sqlite > insert into t values ( "Math" , "Alice" , 99 )…
(Updated October, 2024.) Python # As of Python 3.7: breakpoint ( ) In older versions of Python (going back to at least 2.4): import pdb pdb.set_trace() NodeJS # require('inspector').open(9229, 'localhost', true); debugger; But if you're in a module context: import ( 'inspector' ) . then ( ( inspector ) => { if ( inspector . url ( ) === undefined ) { inspector . open ( 9229 , 'localhost' ) ;…
"Docker Container on a VMWare instance on AWS is our version of Turducken." — @cloud_opinion Let's imagine my Mac running VMware Fusion. Inside that, I'm running Ubuntu. In there, I've got a Docker container running a CentOS 6.4 base image. Once there, I've used virtualenv to create a Python environment with my favorite version of Flask. (Or maybe I chose Java, then Tomcat, running multiple WARs!)…
I talked debugging distributed systems at Strata today. Thank you to all those who came! Slides :
I'm talking on Wednesday at Strata about Tips and Tricks for Debugging Distributed Systems . You should come check it out. As a preview, let's talk about two pretty pictures. Network Visualization # I'm running some typical distributed systems (HDFS, MapReduce, Impala, HBase, Zookeeper) on a small, seven-node cluster. The diagram above has individual processes and the TCP connections they've…
I'm speaking at Strata about this stuff in February, so this and a few other posts are by way of preparation for that talk. If you're building distributed systems, expose information about your system over HTTP. I learned this trick at Google, and I've used it in every system I've helped build since. Expose, on an HTTP server inside of your process, as much useful information as possible. Do this…