Recent content in Justin Abrahms - Principal Engineer & Technical Leader on Justin Abrahms
In ~2023, I was part of a team at eBay called the Velocity Initiative. Our charter was to “make software delivery a competitive advantage” for the company. It was very successful on the metrics and less so on the culture. My part was to link up with the ~700 payment engineers to improve their delivery practices. Throughout that process, we talked about many of the common things that…
When I was at Thrive Market, we did a massive technical debt paydown (think: no features for a quarter). Leading into that, I led an all-day workshop for engineering management and tech leads around how we get a handle on the technical debt the company had accreted over time. For context, at the time the company was on a legacy code base in PHP, primarily built around a PHP ecommerce framework…
Ghostty is a fast, native terminal emulator built by Mitchell Hashimoto. I switched to it for its performance and clean defaults. Configuration ( ~/.config/ghostty/config ) # keybind = cmd+period=prompt_tab_title
I’ve worked for globocorps like Amazon, eBay, Walmart and Google. I’ve worked for startups you’ve certainly never heard of.. but nothing in between. Thrive Market, an online grocer, is the mediumest company I’ve ever worked for. This company was my very first experience with “medium company problems”. At a startup, efficiency numbers largely don’t matter.…
MCPs (Model Context Protocol servers) are services that expose tools for LLMs to use. CLIs (Command Line Interfaces) are shell utilities that programmers use to interact with a tool without using something like a website. Both of these tools are available to many agentic LLMs like Claude Code. Which should you pick? (Related: a personal AI agent I built in 2,000 lines of Go , where this question…
The problem # At Thrive Market in 2024, we had ~150 flaky tests and a 76% deploy success rate. This meant that one in four deploys failed. Not because the code was wrong, but because the pipeline couldn’t be trusted. A one line code review took two days to land. Engineers were watching CI fail over and over and retrying ad nauseam. Ephemeral PR environments took 30 minutes to spin up,…
Last weekend, I spent time setting up OpenClaw . Within 4 hours, I had hit $20 in usage (thank you, spend caps!). It was clear that if I wanted to experiment with this type of software, my clanker would need to run within the bounds of subscription pricing. For the purpose of this post, I’ll refer to the software repo I’m using ( epiphyte ) as my “claw” and the instance of…
I was looking into embeddings recently and realized you can embed your personal knowledge base and a book’s sections/chapters/pages, then use cosine similarity to estimate what you’re likely to learn. mindblown.gif The result is the ability to diff a book against your current knowledge base. This lets you glance at a technical book and see whether you’ll learn something new, deepen your…
When I lived in NYC, I frequented hack && tell , a hackers’ meetup where folks showed off side projects and generally were nerds with one another. During that time, I was trying to better understand how to show up for my friends and community. I came up with what some might call a technical solution to a social problem, but, after 15 years of reflection, I’d say it’s a bit closer…
I came across this post on social media about how a user was using OpenClaw to generate tiktok content for their iOS app automatically. This made me realize that we could be using LLMs to generate product demo videos of the features we work on. I worked with Claude in order to generate on for a flagd UI I was playing with, slapped some royalty free music on it and here was the result.
There’s a common pattern of building up a council of reviewers and have them debate a topic. The first ones of these I saw was the Council of HAL . Following that, I’ve seen similar things like with open-code-review , which will bring up a panel of experts and have them debate, then synthesize the feedback and write it into a PR. In doing some design doc review, I stumbled on a pattern…
AI skeptics complain about code which is verbose, overwrought, unmaintainable, and misguided. In many specific instances, I’ve found that to be correct. The problem to be solved isn’t higher ownership, more diligent human review or the perfect CLAUDE.md file. We need better control over the inputs if we want to tighten the band of acceptable outputs. The core problem with LLMs going…
When running multi-agent orchestration like gastown or multiclaude , there’s a lot of moving pieces. It can be noisy to determine what’s really going on, especially as it’s happening across several tmux sessions. To address this, I’ve released agent-chat . It shows communication between the agents as though you were in a chat room with them. One chat-room per…
Dan Lorenc recently started talking about his LLM orchestration experiment dubbed multiclaude . I’ve used it a bit and thought others who were looking at it might want to know how it’s different. For the comparison point, I previously wrote up Gas Town , the larger orchestrator this is reacting to. How it feels # Multiclaude is similar to Gastown in that it’s a distributed go…
What follows is from an LLM session w/ Claude Code, where I was discussing some of the tensions in the current world. This accurately represents where I think things are heading. Is this a perfect analogy? Absolutely not. But it’s directionally interesting and indicative of the levels of change I think we’re looking at in the next 1-3 years. A Tuesday in 2027 You open your morning…
In Welcome to Gas Town , Steve Yegge generated a list of the 8 stages of agentic workflow evolution. I find myself wanting to deep link to it, so I’m replicating it here. If you want the tool itself rather than the model, I’ve written up what Gas Town is and how it works . Stage 1: Zero or Near-Zero AI: maybe code completions, sometimes ask Chat questions
I read Steve Yegge’s post announcing Gas Town , an LLM orchestrator which allows you to manage dozens of Claude Code instances at once while they make independent progress towards some stated goals. It was a wild ride, but I believe I see the promise. Context # Two pieces of context seem useful before diving in. First, I worked with Steve briefly at Google. I remember him working on…
Trying to learn how to use LLMs is a pain. So many of the day-to-day invocations are hard to pick up. Pairing with people isn’t very scalable. To that end, I saw a slack message by David Edmiston about a tool he wrote: Trivial, but in OpenCode, just one-shotted (and then tweaked the result a bit) a single page utility that can take a Claude conversation (JSON file exported from the web…
At work, I rolled out backstage as a service catalog offering in 2025 across a few hundred git repos. It provided a pathway to get rid of a spreadsheet hand-maintained by a developer about what each repo or service was, who owned it, what it depended on, etc. When our regular compliance reviews came around, it made me think: Can this help us here too? It turns out it could.
Problem: The cluster is in a broken state because consul can’t seem to gather a quorum w/ it’s raft implementation. In my case, there was a raft peer that was bogus. I accidentally had it advertising it’s IP as 127.0.0.1 , but there was no process who had that node-id at that address. There are two possible paths out that I know of. You can put a peers.json file in the consul…
This sets up my local nix workflow. It uses home manager . { config , pkgs , ... }: { # Home Manager needs a bit of information about you and the paths it should # manage. home . username = 'abrahms' ; home . homeDirectory = '/home/abrahms' ; xdg . dataHome = '/home/abrahms/.local/share/' ; nixpkgs . config . allowUnfree = true ; # This value determines the Home Manager release that your…
On it’s face, using open source code is an inherently risky endeavor. We are trusting external developers to write code that we’ll eventually deliver to our users, and we have no recourse if the code is buggy or malicious in some way. In practice, people are generally good and the forces that govern popularity of open source projects help reduce this risk to the point where nearly all…
Last week, I attended SREcon 2023. I was invited to attend the “next-gen delivery” working group meetings, due to my involvement within the Continuous Delivery Foundation (CDF). I used the opportunity to attend the rest of the conference as well. While I’ve never been an SRE by title, I’ve toed the line between operations and software development for many years. Conference…
As part of the OpenFeature project, I’ve been thinking a bunch about feature flags. There’s been ambiguity about how feature flags differ from dynamic config and if that’s the same thing as experimentation (e.g. A * B testing). At their core, feature flags are really fancy if statements. Those if statements control the behavior we want to manage for our application. // example…
Etsy published a recent article which documented how they migrated from their old search service to one which was based on deep learning. The majority of that article made little sense to me, so I wanted to take a moment to document what the heck that actually meant. Originally, Etsy’s search service was based on a gradient-boosted decision tree model, which drove personalized search. They…
I host this blog on a git repo that lives on the same box as the webserver. When I push to it, I want to ensure that deploys happen. Previously, it involved SSHing and doing a bit of a dance. Today, I setup a deploy-on-push script through git’s pre-receive hooks. It was surprisingly difficult to find a good example of this pattern, so I wanted to publish my result. When the pre-receive hook…
I recently read the article Zettelkasten/Org-roam/Org-brain Is Crap . After sending this response to the author, they asked for a place to link it. There were a few misconceptions to clear up and a few tips I thought I’d share. For context, I went from a very large eng-log.org file to multiple files (~210) managed through org-roam at the end of 2021. I probably don’t follow zk…
The Continuous Delivery Foundation is currently looking to build out a reference architecture, which I think is a fantastic idea. While there are a bunch of social things that need to be figured out to really “get” CI/CD, the [[ https://bestpractices.cd.foundation/ ][Best Practices SIG]] is working to get those well documented. I thought it might be helpful for me to document what my…
It’s 2022. 1 bitcoin is about $35.5k USD. An ethereum is $2.3kUSD. I hold 0.2ETH, which I bought because I really like Gary Vaynerchuk, and I was interested in his NFT project VeeFriends . Ultimately, those NFTs were too expensive for me to feel comfortable participating, primarily around gas fees. I first heard about bitcoin in late 2010 or early 2011 at hack&&tell . A finance person was…
I was working on ox-gemini, a gemini exporter for org-mode documents. In the latest versions of emacs, string-replace is a function which replaces a word in a string. It doesn’t exist in emacs 27.2 and earlier, so I added a simple shim. ;; backport for older emacs verisons ( if ( not ( fboundp 'string-replace )) ( defun string-replace ( from to in ) ( replace-regexp-in-string ( regexp-quote…
In the enterprises I’ve recently been a part of, there’s been lots of discussion about “governance”. Governance is a process where you ask other people for permission to do things. The governing body serves to enforce consistency in things like API design or to ensure adequate testing on deploys to production. One of the biggest problems with governance is that the…
There are several differences between the traditional REST API model and how federated GraphQL operates, which can cause some friction when discussing the tech with your SRE/ops organization. This is a list of issues that I’ve come across in my work and the mitigations we’ve either considered or adopted. This list assumes that you either have access to Apollo Studio or have replicated…
Note: This was originally posted internally at Walmart, and has since been sanitized for public consumption. The postmortem process is a tool that we use to better understand failures within our systems. There are two ways to view failures within complex systems: “That failure cost us $250,000” or “The company spent $ 250,000 to learn this lesson”. Taking the second…
As part of our efforts of continual improvement, we have been diving deep on the metrics that we want to track at an organizational level. This has been happening at all levels of the organization, which has resulted in multiple contemporary conversations around which metrics we should track. In these discussions with engineers and leaders, we’ve lacked some wording to classify the metrics we were…
A biogas digestion system is a way to convert biological waste (like kitchen scraps, grass clippings, and manure) into usable biproducts of methane and both liquid and solid fertilizers. Biogas digestion works by creating an anerobic chamber (i.e. a place without oxygen) which breaks down the organic matter you put in it. There are two types of bacteria at play here. One type takes food and…
Someone recently asked: “Is web scraping an ethical concept?” I believe that web scraping is absolutely an ethical concept. Web scraping (or screen scraping) is a mechanism to have a computer read a website. There is absolutely no technical difference between an automated computer viewing a website and a human-driven computer viewing a website. Furthermore, if done correctly, scraping…
Microsoft released a new font called Cascadia Code. I’d prefer if I could install this with the debian package manager, so I went to figure out how to put that on Ubuntu’s PPAs. # Install dependencies for the ppa creation process sudo apt install -y dh-make bzr-builddeb # Tell bzr who I am bzr whoami 'Justin Abrahms <justin@abrah.ms>' # Download the font wget…
Six people live in my house. We don’t have strong technology overlap (some folks in the house only use simple phones and others aren’t keen on things like slack or similar). We’ve addressed this by centralizing around a large whiteboard in our kitchen space. This holds things like grocery lists, house announcements, and similar. I often do grocery shopping on my way home, but…
Serene lake is a backpacking trip that I’ve done a few times. It features wonderful Oregon moss-y forests, a variety mushrooms, huckleberries, and some wonderful lakes. If you’re lucky, you can also catch some mountain views from ontop of the hike’s one ridgeline. The hike is lollipop shaped from the entrance and clocks in at around 14 miles. It starts at Shell Rock lake. There…
Quality Assurance (or QA) is a mechanism by which teams confirm that they’re hitting the requirements they’ve said they’d hit. One example is ensuring you’re validating your contractual obligations from a consulting standpoint. Another is making sure you meet the end user’s expectations. A friend who does QA in the nuclear industry recently had some issues with an outside software vendor. They…
I do a yearly bike ride out to LL Stub Stewart State Park, between Banks, OR and Venonia, OR. It’s a fantastic intro to bike camping trip that’s a known quantity. The full ride is around 48 miles and requires minimal meal preparation given it’s access to food and grocery stores. The guidance I give to my friends is: “If you bicycle around town to get around, you can do the…
This is the third in a three post series. The first post explains Big-O from a self-taught programmer’s perspective . The second post talks about how to calculate Big-O . In my original Big-O piece , I’ve made a bunch of simplifying assumptions and told you a bunch of half truths. It might seem a bit disingenuous, but really.. these half-truths are helpful because (unless you’re going after…
One of the early classes in computer science curriculum is “data structures and algorithms”. When you read computer science text books, you’ll see these words paired together. This makes the word sound more fancy and important than it should. An algorithm, while important, is not particularly fancy. Algorithms, simply put, are how you go about doing something. When paired with data structures,…
@justinabrahms serious question: will binary always be the lowest level computer language? Is there (conceivably) a different model? — Justin Jackson (@mijustin) December 15, 2015 Justin Jackson asked if binary will always be the lowest level language, or if that might ever change. To answer that, let’s talk about what low level means. Binary isn’t exactly the lowest level programming…
Google’s Guice is a Java-based dependency injection framework, which means approximately nothing to people who aren’t familiar with it. Google’s Guice is a way to build a graph of dependencies so you can instantiate complex objects made of simpler parts. Google’s Guice is a replacement for the new keyword for many of your objects. Bizzarre, right? In a more tangible sense,…
I’ve come across multiple neat tools. Given all of my configs are literate programming configs, this sets up those scripts. Otherwise, I’d either not have those tools, or I’d have two processes for getting my dotfiles setup. snapshot ( ~/bin/snapshot ) # A simple screenshot tool. Uses screencapture on macOS or imagemagick’s import on Linux. #!/bin/bash name = $1 if [[ '…
I’ve just started my latest project at MIT, and I’m quite excited about a new piece of technology that the team uses: apiary.io. With Apiary, we generate a simple blueprint file ( source , output ) which is a full description of your API. This shows off requests and responses with a bit of prose. It’s all in markdown file and is very simple to get started. There is a tool called…
I’ve recently accepted a job at MIT’s Office of Digital Learning which I’m quite excited about. The bulk of the work is done in open source on GitHub. We’re working on a stack I’m familiar with (Python/Django) but are also actively using newer technology I’m interested in (React/Docker). Technology choice wasn’t the actual reason I chose this job though.…
I’ve just returned from MicroConf 2015. MicroConf , if you’re unfamiliar, is a conference for self-backed startup entrepeneurs. Most of the talks center around how to grow out a small business. The audience is split roughly evenly between consultants and product. There were two stand-out talks for me this year. One by Patrick McKenzie and the other by Steli Efti. The first talk of the…
I recently sat down to write a node service and happened upon a very helpful workflow when trying to write tests that I wanted to share. When writing tests, one of the biggest jobs is to ensure you have a reasonable degree of coverage on the various branches in your code. This means that if you have an if statement, you want to check the case where the if is true and the case where the if is…