Running Migrations Once During a Multi-Host Kamal Deploy

At my day job, we deploy our Rails app in production to three separate web hosts using Kamal. Each container runs bin/docker-entrypoint on startup, which runs bin/rails db:prepare. That worked fine when there was only one host (like staging), but ran into issues with many hosts. All containers start at roughly the same moment, all three try to apply pending migrations, and all three race for the same migration advisory lock.

One of them wins. The other two see the lock is held and bail out with ActiveRecord::ConcurrentMigrationError. If migrations finish in milliseconds, you might get lucky. Ours sometimes take a few seconds, which is plenty of time for the "losers" to give up and crash the boot.

The fix is small and lives in two places: a Kamal pre-deploy hook that runs db:prepare exactly once, and a slimmed-down bin/docker-entrypoint that no longer tries to migrate at all.

Moving migrations into a pre-deploy hook

Kamal will execute any executable script in .kamal/hooks/ at the appropriate point in its lifecycle. The pre-deploy hook runs after a new image has been pulled from the registry and before any containers are restarted. That's the window... the new code is available, but the old containers are still serving traffic.

Here's the full hook:

#!/bin/bash
set -euo pipefail

if [ "${KAMAL_COMMAND:-}" = "rollback" ]; then
  echo "Skipping db:prepare on rollback"
  exit 0
fi

if [ -z "${KAMAL_VERSION:-}" ]; then
  echo "KAMAL_VERSION not set, refusing to continue without running db:prepare" >&2
  exit 1
fi

if [ -z "${KAMAL_DESTINATION:-}" ]; then
  echo "KAMAL_DESTINATION not set; refusing to run db:prepare against base config" >&2
  exit 1
fi

dest_flag=(-d "$KAMAL_DESTINATION")

echo "Running db:prepare on primary host against $KAMAL_VERSION (${KAMAL_DESTINATION:-default})..."
exec bin/kamal app exec "${dest_flag[@]}" --primary --version="$KAMAL_VERSION" "bin/rails db:prepare"

The "magic" is this command: exec bin/kamal app exec "${dest_flag[@]}" --primary --version="$KAMAL_VERSION" "bin/rails db:prepare"...

The --primary flag tells Kamal to run the command on a single host (the first one in the deploy config) instead of all of them. So we run migrations on only one host.

The --version flag pins the command to the new image we just pull. Without it, Kamal would run against whatever is currently the "live" image on that host, which is the old code.

The -d flag forwards the destination (staging, demo, production, etc.) via the KAMAL_DESTINATION environment variable.

We bail loudly if KAMAL_VERSION or KAMAL_DESTINATION is missing. They should always be set, so if the script fails because of that, something went terribly wrong.

Slimming down the entrypoint

With migrations handled by the hook, bin/docker-entrypoint no longer needs to run them on container boot. The following block of code is removed from the file:

# If running the rails server then create or migrate existing database
if [ "${@: -2:1}" == "./bin/rails" ] && [ "${@: -1:1}" == "server" ]; then
  ./bin/rails db:prepare
fi

A small side quest: bin/kamal

Kamal's hooks run on the deployer's machine, not inside a container. To call bin/kamal app exec from the hook, we need bin/kamal to be runnable in whatever environment the deploy is happening in. If that environment doesn't have all of our app's gems installed (ie - developers using dev containers), the original require "bundler/setup" will fail trying to resolve the entire Gemfile.

We don't need the full bundle to run kamal. We just need kamal. So bin/kamal now reads the kamal version out of Gemfile.lock and uses bundler/inline to install just that one gem on the fly:

require "rubygems"
require "bundler"

kamal_version = Bundler::LockfileParser
  .new(File.read(Bundler.default_lockfile))
  .specs
  .find { |s| s.name == "kamal" }
  &.version
  &.to_s || abort("kamal not found in Gemfile.lock")

require "bundler/inline"

gemfile do
  source "https://rubygems.org"
  gem "kamal", kamal_version, require: false
end

load Gem.bin_path("kamal", "kamal")

Pulling the version directly out of the lockfile means the binstub can't drift from whatever version we're actually deploying with.

And we're done

Three changes, one annoying bug killed. Migrations now run once per deploy on a single host against the new image, the entrypoint is shorter, and bin/kamal no longer demands a fully-installed bundle to do its job.