Recent content on John Tobin
Hi, I’m John Tobin. Since July 2010 I’ve been working in Site Reliability Engineering (SRE) at Google . Currently I’m the manager of Bigtable SRE in Dublin; previously I was an SRE running Bigtable, Colossus , and GFS . I’m a double graduate of Trinity College, Dublin : I completed a B.A. (Mod) in Computer Science in 2002, and an M.Sc. in Computer Science in 2009; my M.Sc.…
2008 Link to heading A User-Extensible and Adaptable Parser Architecture , published in the SGAI 2008 Conference , and selected as one of the best papers from the conference. I was subsequently asked to join the Technical Stream Program Committee , and I accepted the invitation. The best papers from the SGAI 2008 Conference were reprinted in a special edition of the Knowledge-Based Systems Journal…
I wrote and maintain the theme for my wife’s website . I installed everything I used on my laptop with these commands: brew install composer php php-code-sniffer php-cs-fixer phplint phpmd phpstan \ phpunit composer require nunomaduro/phpinsights --dev pecl install xdebug Testing Link to heading For testing I’m using PHPUnit with CodeCoverage enabled (code coverage also requires Xdebug…
Motivation Link to heading I recently switched to Neovim after over 20 years of using Vim. (At some point I might write about why I switched, but not today.) One of the few features I missed in Vim compared to an IDE was debugging: sure, I could fire up dlv or gdb , but I didn’t have syntax highlighted code, automatic display of local variables, and likely other helpful info I didn’t…
I’m slowly learning Rust , and as usual I’m using Vim to write my code. I’m using rust.vim for syntax highlighting, and vim-lsp and rust-analyzer for Language Server Protocol support. As I wrote more code I felt that saving my source file was getting slower, sometimes taking up to 10 seconds! Eventually this got annoying enough that I used Vim’s profiling support ( :help…
Like most Irish households, we aggregate our household waste into wheelie bins that are collected every two weeks. Ours are collected on Monday morning, so they need to be left out on Sunday night - except when Monday is a public holiday, they’re collected on Saturday, and so they need to be left out on Friday night. This has worked for years, but recently I forgot to put out the wheelie…
By default, when Homebrew upgrades a package, it removes the older version. This is generally the right thing to do, but it will badly affect some running programs, e.g. tmux and vim , because the binary no longer exists. tmux cannot create a new window, and vim cannot run any external programs. The error message from vim is very confusing and it took me a lot of debugging to figure out the…
Humans make mistakes - that’s why we write tests for our code. Automatically running tests or other checks when committing code is a useful way to catch mistakes and prevent committing bad code, and Git provides many different hooks that can be used to automatically run such checks. Initially I wrote hooks as shell scripts, but recently I checked out https://pre-commit.com/ and it both makes…
Normally I do Python development in work, where everything is already set up for easy development and testing. It took considerable research and setup to get a similarly good environment for Python development at home, so I decided to document it for myself and others :) Install all the packages Link to heading Mac OS Link to heading The list of Python modules you need is: black lxml mutmut mypy…
Your dotfiles are the project that you’ll work on for longest. I spend a lot of my time using Vim, Bash, and various CLI tools. Over the past 15 years I’ve spent a lot of time configuring these tools, and I’ve gotten so used to my configuration that it’s really weird when I don’t have it. I use 6 machines on a regular basis (some with multiple accounts), so I need a…
In work we have tooling that automatically runs tests when you save a file from your editor, saving you from pressing up arrow, return to run them manually. I decided to implement that for use when I’m coding in my own time: inotify_wrapper . Usage is straightforward: Watch the specified files and run the binary when they change. Usage: inotify_wrapper file1 [file2 ...] -- binary [args ...]…
Photos Link to heading More photos and three short videos! Resources Link to heading How to Paint: Lord of Contagion Example Death Guard on GW HOW TO PAINT: Plague Marine Reinforcements How to Paint: A Pallid Hand Plague Marine - completely different colours for the armour, but the Nurgle tentacle parts might be useful. How to Paint Death Guard: Nurglings Casandora Yellow looks like a good…
I write a lot of Markdown in work, so it’s worth configuring vim to support it well. Install https://github.com/preservim/vim-markdown to get syntax highlighting. I disable that plugin’s folding in favour of my own because I find mine is simpler and works better. Install https://github.com/google/vim-codefmt and https://prettier.io/ to get autoformatting on write - you’ll need…
There’s a saying that if all you have is a hammer, everything looks like a nail ; I tend to take the SRE approach to everything, including gardening, so here are some thoughts about gardening. My context is that we have a relatively large garden, and I don’t like gardening; if you have a more manageable garden or you like gardening, you might not find this helpful. Reduce your toil…
For many years I’ve used Version-controlled /etc on Linux to track changes. On MacOS there isn’t a location in /etc to put a crontab , and my user crontab is in /var/at/tabs/ , so how can I keep it version-controlled? I solved this problem by writing a wrapper program: crontab-edit . When run without arguments it sets $EDITOR so that crontab -e runs crontab-edit TEMPORARY_FILE . When…
I have a VM from Hetzner for my wife’s website and my own. I run several probers hourly from cron on the VM and on a machine in my house (so they aren’t singly homed) to ensure that the VM, DNS, Apache, and websites are working properly: The simple link checker I wrote checks my wife’s website, the development version of my wife’s website, and my website. Checking the…
Simple link checking describes how I check internal links, but what about checking external links? You need to traverse every page on your website to find the external links to check, but you definitely must not traverse every page on external sites! I found a tool that supports this: linkchecker . (Note that development seems to have stopped there and moved to a new group , but I’ve linked…
I use rsync.net for offsite backups with history (extended to 21 days of snapshots). The simple way to use it would be to have a single ssh key without a passphrase that all machines backing up use, and run rsync once for each directory being backed up. I didn’t take that approach because that single ssh key would give access to all our backups if one of our laptops was stolen; instead I…
This blog post describes the overall backup strategy and implementation for my personal machines. Each machine backs up everything except photos to rsync.net , providing offsite backups with history; photos are excluded because of their sheer size. See backing up to rsync.net for implementation details. Each Mac uses Google Backup and Sync to back up all data including photos, for offsite backups…
Ikiwiki to Hugo Link to heading I migrated my website from Ikiwiki to Hugo . I had been using Ikiwiki since 2009 but a few things annoyed me about it: It’s not packaged for MacOS, which means I have to write on my hosting instead of my laptop, and my slow internet access makes that a bit painful. Because I’m editing remotely previewing changes is awkward and is not a good…
There are several apps I run in the background on my laptop that occasionally crash, e.g. Google Drive for Desktop . After several crashes that I didn’t notice for a few days each, I’d had enough and decided I needed tooling to automatically restart certain apps. I searched for existing tools but only found some blog posts about the problem, so I wrote my own. The implementation is…
New versions of Wordpress are released regularly, and themes and plugins are upgraded frequently. I wrote wordpress-install to make updating a single plugin/theme/Wordpress easy, but it still requires running a command like wordpress-install DIRECTORY theme twentynineteen for each plugin/theme/Wordpress that needs to be updated. That’s quite toilsome so I wrote a simple wrapper around…
Because I work in the tech industry, family and friends will sometimes ask me to recommend a laptop for them to buy. In the past I’ve done the research myself, but recently I’ve been empowering people by sending them instructions to follow, which also saves me time ;) I decided to publish the instructions because they might be useful to others. Figure out your budget, that’s the…
The EU Cookie Law requires a notice about cookies on every website. My wife’s website is built on Wordpress , so to display the cookie notice I use the GDPR Cookie Consent plugin. The CSS and JavaScript resources the plugin uses are unconditionally added to every page, but they are really only needed on the first page load: the user accepts the cookies if they stay using the website, so the…
I use Apple’s Time Machine to make local backups of our home computers. It works really well with a locally attached disk, but it fails often when backing up over the network, e.g. because the network is slightly flaky, the machine backing up goes asleep, or something else. After a backup fails Time Machine will try to validate the backup volume, but there’s a high chance that it is…
I wrote and maintain the theme for my wife’s website and I maintain her website in general. I have a development version of the website so that I can develop the theme and test plugins without breaking the production site. For several years I had only a small number of pages in the development website, each testing some combination of features, but they weren’t properly representative…
My wife’s website is built on Wordpress . Wordpress uses a MySQL (or MariaDB ) database to store pages, posts, comments, sessions, and everything else except media. That database needs to be backed up regularly in case of data loss. Backing up a MySQL database is easy - just run mysqldump - so why would I need anything else? I wrote backup-wordpress to wrap some extra functionality around…
My wife’s website is built on Wordpress . It uses a small number of plugins, and I’ve tested many plugins over the years; all those plugins need to be installed, and of course Wordpress core, plugins, and themes need to be upgraded when new versions are released. This isn’t difficult (download core/plugin/theme, extract zip file in the correct directory) but it’s tedious so…
I use Git for many things, and I have written several utilities that wrap git commands (e.g. pushing changes to the theme for my wife’s website from the dev site to the production site ). A common requirement is that the local copy is clean : no uncommitted changes, and no unpushed commits. I had been checking for uncommitted changes by testing if there is any output from git status , but in…
I’ve been annoyed recently by MacOS not automatically reconnecting to WiFi when our router is restarted. This is slightly annoying when it’s my laptop, but it’s really annoying when it’s the Mac Mini, because I have to grab a keyboard and manually reconnect. I decided that I should automate this away, and after some searching I found networksetup . I wrapped that with a…
Note: this was originally a lot longer and more complex, but a later version of tmux show-environment supports formatting the output as shell commands to eval, so this is much easier now. tmux is a tty multiplexer similar to screen , but with some really nice features. One of those features is updating environment variables when you reconnect to a session - the client sends the current values to…
I’ve been working on my wife’s website recently, and I wanted to check that all the internal links and resources worked properly. I wasn’t going to do this by hand, so I wrote a simple wrapper around wget . It deliberately downloads everything and saves it to make finding the location of broken links easier. Any request that wasn’t answered with HTTP status 200 is…
For several years I had a Subversion repository named cs_misc , where I accumulated various pieces of code that didn’t need a repository of their own. A year ago, I decided to switch to Git, and created a repository named cs-misc . As described in Importing Subversion repositories to Git , I migrated cs_misc from Subversion to Git, and now I wanted to merge the two repositories ( cs-misc and…
I’m migrating all my source code repositories from Subversion to Git. I tried git-svnimport , but it only works if your repository has the recommended layout of trunk , tags , and branches ; unfortunately, a lot of mine don’t. git-svn initially looked like overkill, but it worked quite well. Below is the simple shell script I used to import my repositories and push them to Github; I…
About a month ago I wrote that I need a better LISP book . I gave up on the LISP book I had been reading, and started reading On LISP: Advanced Techniques for Common LISP by Paul Graham. I’ve read about one third of it, and understood most of it - I had trouble with some of the more difficult code, but I understood his explanations of LISP features and what the code was doing. I was…
If your website is available under more than one FQDN, standard SEO advice is to pick a canonical FQDN and redirect the others to it. You can see that in action on this website: johntobin.ie/blog/smarter_http_redirects will redirect you to https://www.johntobin.ie/blog/smarter_http_redirects/ . The simplest way to do this in Apache is to configure a VirtualHost for johntobin.ie, and use a single…
I’ve been learning LISP from LISP , and solving problems from Project Euler in LISP. Once you solve a problem, you gain access to the forum thread about that problem. After I solved Project Euler 10, I read someone else’s LISP solution; it was quite different to mine, and contained some LISP constructs I hadn’t seen before, e.g. ( defun seq-list ( min max ) ( loop for i from min…
In 2009 I migrated the School of Computer Science and Statistics mail server from Solaris to Debian Linux. I made a lot of changes and improvements during the migration; one of the simplest was to keep /etc under version control. I assume most people are familiar with version control from writing code - if you’re not, please spend a couple of hours reading and experimenting with any modern…
From https://www.ntppool.org/en/ : The pool.ntp.org project is a big virtual cluster of timeservers providing reliable easy to use NTP service for millions of clients. The pool is being used by millions or tens of millions of systems around the world. It’s the default “time server” for most of the major Linux distributions and many networked appliances (see information for…
I first thought about learning LISP when I was still an undergrad, but I was stymied by Real Life and a lack of material to learn from. Shortly before I submitted my MSc thesis I picked up two LISP books - LISP and On LISP: Advanced Techniques for Common LISP - but my MSc was taking up all my time, so I put them on a shelf and forgot about them. About a month ago, I read Recursive Functions of…