Recent content on David Winterbottom
This post details a Bash script that runs a TDD loop where an LLM is iteratively used to write Python application code to make a fixed set of tests pass.
Similar things have been done before1; this script merely demonstrates that such things can be done simply by stitching together open-source LLM tools.
Tooling The script uses two commands from Simon Willison:
llm — a command line…
I write a lot of Markdown in Vim and have spent considerable energy configuring it to my liking. This post details how I configure Vim1 for writing markdown.
It is merely a reference that I can refer others to.
File-type settings In ~/.vim/after/ftplugin/markdown.vim (or equivalent), set some buffer-local settings:
' Use Vim's spell checker setlocal spell ' No line numbers setlocal…
Based on a few weeks of using vim-copilot, I recommend the following:
Enable Copilot for the gitcommit, markdown and yaml filetypes:
let g:copilot_filetypes = { \ 'gitcommit': v:true, \ 'markdown': v:true, \ 'yaml': v:true \ } By default, these and few others are disabled but I’ve found them to be useful. It’s often amusing to see Copilot’s attempts to complete your…
A friend of mine has been retweeting great paintings from Twitter accounts like @HenryRothwell, which I’ve greatly enjoyed. E.g.
Good morning - I hope you slept like a caddis-fly larvae-stuffed miller's thumb. I'm starting with 'Winter Landscape', Valerius de Saedeleer, oil on canvas, 1930s. pic.twitter.com/hk6RUFh794
— Henry Rothwell (@HenryRothwell) December 18, 2022 In a…
I’ve created a deeply middle-class Git scraper project which tracks the prices of a basket of goods sold by the British online supermarket, Ocado.
For example, Lurpak butter:
I’ve been looked for an excuse to use Git scraping for ages, and this idea came up as my wife and I were commiserating over how much food prices are increasing at the moment.
The project is the…
OpenAI provides a REST API where you can generate prompt completions. Here’s a minimal example where a JSON payload is piped into httpie:
$ export OPENAI_API_KEY='...' # fill in your API key here $ echo '{'model': 'text-davinci-002', 'prompt': 'Write a poem about cheese'}' \ | http https://api.openai.com/v1/completions Authorization:'Bearer $OPENAI_API_KEY' \ | jq -r '.choices[0].text…
From the final novella of David Mitchell’s The Bone Clocks, set in Ireland in 2043:
‘Number one is to survive’, answers Hood, watching the men on the roof. ‘They’re all dead, like my parents. They had a better life than I did, mind. So did you. Your power stations, your cars, your creature comforts. Well, you lived too long. The bill’s due. Today,’ up…
As a 1Password admin, there are common audit questions you need to answer around who created various resources. The web dashboard is excellent but some questions are still fiddly to answer. Questions like:
Who created this group? Who created this vault? Who created this item? Who has access to this item? This post is a note-to-self on how to find these answers1.
Who created this group? If…
For the record, I wrote up some patterns that cause flakey tests on the Kraken Technologies tech blog.
Someone has asked a question (in Slack or Github) and you’re about to write an explanation. But before you start typing, ask yourself this:
Is this the best place to answer this question?
Because the place where the question is asked is generally not the best place to write a detailed answer.
“Why did you do it that way?” Imagine you’ve requested a review on a…
A good understanding of Vim’s various lists is a massive productivity boost — it’s taken me many years of Vim use to truly appreciate this.
This post summarises some of Vim’s lists, detailing their purpose and how to make the most of them.
Contents:
Quickfix list Tip: Use mappings for faster browsing Tip: Define a mapping to :grep for the word under the cursor Tip:…
Here’s a useful command-mode mapping for Python development:
' ~/.vim/ftplugin/python.vim function! VirtualEnvSitePackagesFolder() ' Try a few candidate Pythons to see which this virtualenv uses. for python in ['python3.7', 'python3.8', 'python3.9'] let candidate = $VIRTUAL_ENV . '/lib/' . python if isdirectory(candidate) return candidate . '/site-packages/' endif endfor return ''…
This is a note-to-self on setting up a 2020 13-inch MacBook Pro, largely for Python development. I imagine it will all be out-of-date by the time I set-up my next laptop but it’s possible it might be useful to someone in the meantime.
Applications I don’t tend to use Homebrew Cask and suffer the indignity of either installing them from the App Store or downloading .dmg files and…
Here’s a useful technique for using Terraform’s dynamic blocks to create conditional nested blocks.
Maintenance mode As an example, let’s create a “maintenance mode” for a service which allows a “under maintenance” holding page to be served when a Terraform variable is set.
This is useful if you need to stop all traffic to a RDS database server so…
Problem Your Terraform config requires managing many CIDRs that control firewall ingress rules. You’ve been storing these in a CSV string:
variable 'client_cidrs' { default='50.1.1.1/32,44.2.1.0/32', } which is fed to a aws_security_group somewhere in your configuration.
The CIDRs change frequently and maintaining this variable is difficult as it’s hard to track where each…
URLs are great aren’t they?
You include them in your Slack messages and your co-workers can see exactly what you’re talking about in a single click. I wish people would use them more (and design apps that support them properly).
Anyway, a super-useful Vim mapping I use is:
vnoremap <leader>gb :GBrowse! master:%<cr> which, after visually selecting a block of code, grabs its…
On code smells:
If your codebase is tightly coupled to data in your database (ie the codebase has a data value hard-coded), it is a sign you should extract that data from your database into a code-layer model.
Never use numbered variable names (eg account1 = ...) – there’s always a better way.
On good things I always tell team members about:
The writing and conference…
Resolving conflicts from a Git rebase can be tricky. But don’t worry – here’s a comprehensive guide to how to resolve them.
There’s three phases:
Which commit of mine is conflicting? What changes were made in the target branch that conflict with my commit? Resolve conflicts safely These are accurate as of Git v2.23 and are for resolving conflicts using the command…
On software development:
Everything you create that has a name lives in a namespace. Remember this. Ensure the names you pick are unique and unambiguous within their namespace.
If a 500 Internal Server Error HTTP response can be induced from your web app through a carefully crafted request, it needs fixing. Don’t assume anything about the incoming request.
On tools:
Using…
Text objects, as in the iw from ciw (“change inner word”), form an important part of your Vim mentalese1. This post details those that I find most useful for Python and Django development.
For brevity, the leading a (mnemonic: “a"n) or i (mnemonic: “inner”), that you combine with the following commands to form the full text-object, are omitted.
From core…
There’s only so far you can get by cargo-culting other people’s ~/.vim folders. An important next step is understanding how to debug Vim; knowing what to do when it’s slow or misbehaving. Learn how to scratch things that itch.
This post illustrates a range of debugging and profiling approaches for Vim by walking through real issues I’ve recently investigated, diagnosed…
I wrote up a lesson on the dangers of renaming the related_name of a Django model field over on the Octopus Energy tech blog.
FYI, the easiest way to get Vim to automatically run black and isort over a Python buffer when saving is to use Ale’s fixer functionality.
' In ~/.vim/after/ftplugin/python.vim (or somewhere like that) let b:ale_fixers = ['black', 'isort'] let b:ale_fix_on_save = 1 If you’re only using black/isort in a subset of your projects, you can enable the b:ale_fix_on_save setting…
When provisioning a virtual machine running Ubuntu 16.04 or later, a common problem if being unable to install packages since another process is holding a lock (eg on /var/lib/dpkg/lock-frontend).
This happens as Ubuntu VMs typically start several package-management programs unattended-upgrades and its associated apt.daily service — on boot, and these will block your provisioning…
I wasted a morning trying to install RabbitMQ v3.7.12 (the latest version as of Feb 2019) on an Ubuntu 18.04 machine using Puppet. This as tricky as:
Only RabbitMQ version 3.6.10 is available from the default repositories; Getting Puppet to install packages from custom locations can be painful. Solution Use these Puppet modules in your Puppetfile:
mod 'computology-packagecloud', '0.3.2'…
I care about writing maintainable software1: code that is a pleasure to work with in the long-term; where new requirements can be accommodated smoothly and swiftly – above all, software that is easy to change.
Memorise that phrase.
I feel we lose sight of this overarching guiding principle and harm our codebases by dogmatically pursuing well-intentioned but proximate goals.
The…
Oddly, you can’t pull a report of all groups from G-Suite like you can for users. The only option is to use the API. Here’s how.
Follow steps 1 and 2 from the quickstart guide but instead of the sample Python script, use this:
from httplib2 import Http from googleapiclient.discovery import build from oauth2client import file, client, tools def main(): _print_all_groups() def…
I spend most of my day reviewing pull requests in Github. These are my working notes on what makes a good PR.
Purpose? It should be clear to the reviewer what change is being made and, crucially, why. So ensure your title and description convey the purpose of the PR. Consider including:
Screenshots — such as snaps or gifs of a new UI, or graphs of the devastating performance…
As I get older and grumpier, I increasingly value clean, uncluttered working environments. I’m sure I’m not the only one, so here’s a few useful practices and shortcuts that help me avoid using the mouse and satisfy my need for productivity micro-optimisation.
They are mainly for macOS users.
Hide everything Adjust your system preferences to automatically hide the Dock…
Joining tables on date and timestamp with timezone fields in Postgres1 needs careful handling because of time zones and daylight-saving time.
To illustrate, assume we have two tables:
t1 which has a field of type date and a foreign-key t2_id to t2 which has a field of timestamp with timezone. We want to build SQL queries that join between these two tables with additional date constraints…
A curated1 collection of words-of-the-day from @qikipedia:
Word of the day: ARSLE - to move backwards.
— Quite Interesting (@qikipedia) May 14, 2017 Word of the day: VERSCHLIMMBESSERN - (German) - to make something worse while attempting to make it better
— Quite Interesting (@qikipedia) July 3, 2017 Word of the day: PAREIDOLIA - the imagined perception of a pattern or…
I wrote up a short tip on using a custom Sentry client.
Because:
You moved everything out of the views.py modules when you heard fat controllers were bad.
And you’ve read that fat models are a good idea1.
You need some extra functionality in a template and this is the easiest way to shoehorn it in. Eg:
<p> The current balance is {{ account.get_balance_via_three_network_calls_lol }} </p> You can quickly solve your current problem…
Two tips:
Fail fast You probably already know that you can force Bash scripts to exit immediately if there’s an error (that is, if any command exits with a non-zero exit code) using:
#!/usr/bin/env bash set -e but it’s even better to use:
set -eu -o pipefail so that the script:
exits on an error (-e, equivalent to -o errexit); exits on an undefined variable (-u,…
There’s two non-obvious things to know when starting to use pgbadger with AWS RDS.
First, set:
log_statement = None Don’t set this to all as the AWS docs suggest.
Further, don’t waste your time trying to add a DB parameter to set log_line_prefix to pgbadger’s recommended value: it’s not possible1. Instead tell pgbadger about the log format that RDS insists…
Using both @mock.patch decorators and py.test fixtures can be confusing as it’s not always clear what order arguments are being injected.
For instance, which of these is right? This:
@mock.patch.object(module, 'collaborator_1') @mock.patch.object(module, 'collaborator_2') def test_something_in_module(collaborator_1, collaborator_2, some_pytest_fixture): pass Or…
“Remember, code is your house, and you have to live in it.” - Michael Feathers 🏠
— Programming Wisdom (@CodeWisdom) April 19, 2017 This is the best metaphor I know for promoting or defending software quality.
We do live in our codebases: shoddy software engineering has direct, easily-visualised analogues from construction and house maintenance.
For instance, I’m sure…
I spend at least 50% of each day reviewing, amended (and occasionally merging) pull requests, adding both commits and comments. As such I often want to quickly jump from a terminal window to the pull request detail page to check previous comments or add new.
Even with the excellent hub git wrapper, there’s no easy way to do this. I can jump to the pull request list page with:
git…
Like many people, I use Google Sheets to quickly create and share tabular data. As well as creating spreadsheets by pasting results generated in psql, I often create reports from JSON files using JQ. This post is a note-to-self on how to do this.
Here’s a command to create a tab-separated report from a JSON events file exported from Loggly:
$ cat loggly_events.json | \ jq -r…
I was interviewed, along with Michael van Tellingen, on the “Podcast.__init__” Python podcast to talk about Oscar.
If your Consul key-value store is structured as:
/ A/ X = 1 Z = 2 Y = 3 C D but you now realise you should have namespaced everything within WEBSERVER/ (or something like that):
/ WEBSERVER/ A/ X = 1 Z = 2 Y = 3 C D then this Bash script will help you migrate:
#!/bin/bash set -e # Exit on error # Emit 'key value' lines for all keys in Consul's KV store keys_and_values() { # Recursively…
In approximate chronological order:
“Sorry I missed stand-up this morning” “I didn’t see that email” “I’ll get back to you shortly” “Sounds easy - should only take a couple of days” “It’s nearly finished” “I didn’t have time to write tests” “We’ll clean this up later” 1…
I’ve migrated this site to Hugo so I can host it on Github pages1.
Hugo is a fast and well thought-out static site generator, written in Golang. It’s easy to learn and has some neat features2 - the trickiest part is understanding the difference between various ways pages are organised: “sections”, “types”, “taxonomies” etc.
The Vim plugin…
I wrote an article on Django, ELB health checks and continuous delivery.
Octopus Energy has a tech blog. I wrote an article on Testing for missing migrations in Django.
Last Wednesday was my last day at Yoyo Wallet. Thursday marked my first day at Octopus Energy.
I’m deeply excited about Octopus. Through innovative use of technology, especially automation and cloud computing, Octopus Energy is going to radically change how people consume and pay for energy in the UK. We’re going to be both cheaper and massively better at customer service than the…
Here’s a useful heuristic for writing better commit messages. Set your commit message template to:
# If applied, this commit will... # Why is this change needed? Prior to this change, # How does it address the issue? This change # Provide links to any relevant tickets, articles or other resources and you’ll be guided into writing concise commit subjects in the imperative mood — a…
I often need to grab information from a Postgres database and paste it into a spreadsheet for sharing with others. Google Sheets needs the pasted data to be tab-separated in order to be correctly split into columns. This isn’t the default behaviour for psql but here’s how to configure psql’s output to get it.
At a psql prompt, switch to unaligned output
=> \a set the…
Cloud computing and immutable infrastructure deployments have changed the way I use SSH. I miss the days when I could run:
ssh app1-prod to jump onto a machine and investigate an issue. This would work as, back in the days of yore, your web servers didn’t change IP address several times a week so I could create a helpful alias in ~/.ssh/config:
Host app1-prod User example_user…
For the record, I no longer maintain commandlinefu.com. I’ve handed over the baton to notable Bay Area celebrity, Jon Hendren (@fart). This is a good thing for the site and its users as I have been unable to do much maintenance in recent times. I look forward to seeing how the site evolves.