Monorepo

I like the idea of monorepos. Perhaps it’s the allure of mystery; I’ve never worked anywhere that has one. Sure, I’ve worked at companies with few source code repositories and companies with big and busy source code repositories but never any company that had literally one monolithic source code repository. I’m fascinated by Piper and the lore surrounding the big monorepos at Google and Facebook. Slack’s webapp repo was big and busy but only monolithic if you excluded infrastructure tools and mobile apps from your worldview.

I have a personal monorepo at home called src that’s cloned to ~/src on all my machines. I have a personal monorepo at work called planetcrowley where I commit all my smartest and dumbest ideas, TODO lists, notes, and prototypes. I commit everything but binaries, push, and don’t stress about backups. These are toys, though. More a hack to avoid naming things than a monorepo of the same kind as Piper et al.

Most articles warning the monorepo-curious like me to stay away basically boil down to two points:

1. A monorepo requires all the same tooling you’ll need if you have many repos.
2. And it requires more tooling to conduct traffic, work with subtrees, track dependencies, and generally operate efficiently.

This tooling is all terribly interesting to me, a guy who both believes in outsourcing undifferentiated heavy lifting and who absolutely loves to take on other people’s undifferentiated heavy lifting. But it’s rare for codebases to get so big that they need to tackle the problems unique to monorepos and yet be owned by companies small enough to want to outsource it. (Thus, there’s not much of a market for Piper-as-a-service.)

At the modest scale of my home and work codebases I don’t have any traffic jams that need conducting, don’t operate on subtrees save for the occasional git filter-branch, don’t have outlandish dependency graphs, and don’t feel bogged down by inefficiencies in CI/CD, etc. In fact, my personal wish list for bringing what I imagine I’d like most about a monorepo into my world of many repos has just one item on it: I want to be able to grep the entire codebase.

With, it turns out, not very much elbow grease, and a codebase that easily fits on a modern laptop, this is no bother. Behold, monsterepo, my solution to this problem for myself:

#!/bin/sh
set -e
usage() {
    printf "Usage: %s [-e \e[4mexclude\e[0m,\e[4m...\e[0m] [-x] \e[4morganization-or-username\e[0m\n" "$(basename "$0")" >&2
    exit "$1"
}
EXCLUDE="" X=""
while [ "$#" -gt 0 ]
do
    case "$1" in
        "-e"|"--exclude") EXCLUDE="$2" shift 2;;
        "-e"*) EXCLUDE="$(echo "$1" | cut -c"3-")" shift;;
        "--exclude="*) EXCLUDE="$(echo "$1" | cut -d"=" -f"2-")" shift;;
        "-x") X="-x" shift;;
        "-h"|"--help") usage 0;;
        *) break;;
    esac
done
ORG_OR_USER="$1" shift
if [ -z "$ORG_OR_USER" -o "$1" ]
then usage 1
fi
TMP="$(mktemp -d)"
trap "rm -f -r \"$TMP\"" EXIT
echo "$EXCLUDE" | tr "," "\n" >"$TMP/exclude.txt"
COUNT=1000 LIMIT=1000
while true
do
    gh repo ls "$ORG_OR_USER" --json "name" --limit "$LIMIT" --no-archived --source |
    jq -e -r '.[].name' >"$TMP/repos.txt"
    COUNT="$(wc -l <"$TMP/repos.txt" | awk '{print $1}')"
    if [ "$COUNT" -lt "$LIMIT" ]
    then break
    fi
    LIMIT="$((LIMIT * 10))"
done
while read REPO
do
    if grep -q "^$REPO\$" "$TMP/exclude.txt"
    then continue
    fi
    if [ -d "$REPO" ]
    then
        sh $X -c "git -C \"$REPO\" remote update"
        if sh $X -c "git -C \"$REPO\" diff --exit-code --no-patch"
        then sh $X -c "git -C \"$REPO\" pull \"origin\" \"$(git -C "$REPO" branch --show-current)\""
        fi
    else
        sh $X -c "git clone \"https://github.com/$ORG_OR_USER/$REPO.git\""
    fi
done <"$TMP/repos.txt"

This is a really stupid program. I’m inlining it into this article as a testament to its stupidity, the throwaway nature of its complexity. It’s trivial. And yet it’s giving me all I really want from a monorepo both at home and at work.

I can’t make atomic changes across many services or a client and its server. With dozens or hundreds of repos cloned locally, I also can’t make atomic changes to the codebases of many services or a client and its server. And while I don’t need to be constantly reminded of this, it can’t hurt to have source code management and change management be structurally aligned.

But I can grep the entire codebase, at least until we get crazy big. Count the win.

There’s one extension I have in mind for monsterepo. I’m a big fan of Go workspaces for local codevelopment of changes to multiple repositories. I use them extensively when iterating on Mergician at home or on Protocol Buffers and the services that send and receive them at work. I think monsterepo could automatically go work use ../$REPO whenever it finds go.mod files to automatically maintain Go workspaces for local development. This might prove annoying, though, if the main branches of any of those dependencies are unstable or move too quickly.

It’s also worth noting, with an eye towards the proper disaster recovery exercise I have penciled in for the spring at work, that having an up-to-date local copy of the entire codebase is a prudent insurance policy against GitHub having a disaster at the same time or source code repository backups being broken.