Extracting directories into their own repositories

Software projects in git repositories (or Version Control Systems - yes there are others) tend to be made up of much more than just 1 file, or even 1 directory. To make functions and functionality easier to find, good software is split up, forming modules or isolated components which can be put together to form user journeys or processes. In some cases, these modules may become useful in their own right, and may benefit from being factored out so they can be reused in other projects.

This is exactly what happened to me with django-tasks. I'm not going to go into the specifics, because it's not relevant for this article. The short version is that 2 of the tasks backends previously in the django-tasks repository benefited from being converted into completely independent packages - installable and deployable in their own right.

#But what about a monorepo?

A friend of mine is a big fan of monorepos, and for good reason (mostly). A monorepo is the idea that rather than having multiple separate repositories, you house your entire software project within a single repository. No need to install a set of internal packages - they're all right there in the repo. The exact opposite of the "micro-service" architecture (fad?) people seem to be loving, for some reason.

Monorepos have a lot of benefits, but a lot of drawbacks too. In my case, separating them out was as much about reuse and code management as it was about the mental model of them being separate projects. Sure, you can manage multiple Python projects from within a single repository, but that would only lead to the same confusion about the relationship which I wanted to avoid.

#Does the history really matter?

To me, yes! Another alternative solution would be to simply duplicate the repository and refactor everything in a single gigantic commit for each of the 2 backends. That has a number of problems:

First, it's messy. The history references files which no longer exist, and that 1 commit changes almost every aspect of the project - even its name. It's the git equivalent of "draw the rest of the owl"

Secondly, it's wasteful. The previous history of the repository is still there, meaning every clone drags with it the entire source history. For small projects, that may not be a problem, but for others it could be a lot of wasted time and unnecessary burning of dinosaur juice.

Finally, it's disingenuous. The repository history inflates the number of commits, activity and even contributors to a project. By extracting out just the parts the project needs, those aspects reflect reality (or at least something very close to it)

It's been a while since I tried extracting out a directory into another repository - and back then I just got someone else to do it. The process is actually the exact opposite of what you might expect: The idea isn't to have a blank repository and pick over the commits you want, perhaps with a little tweaking. Instead, you start with the entire repository, and rewrite its history so that the bits you don't want go away.

As with most things in life, it's not quite as simple as "just extract the directory and make it a new package". There are couple of steps to think about before starting.

#Renaming

Chances are, the code you're trying to extract isn't at the top level of the repository, exactly where it would be in its own project. That means, you'll need to rename files as they're processed, rather than just dropping a few commits.

For example, django_tasks/backends/database/models.py needed to become django_tasks_db/models.py, in a way so that the history showed it had always been there.

#Refactoring

When rewriting the history, it's more than just a single file I need to keep. Some of the functionality is elsewhere, in utility modules or helpers. These will still exist, but need to be pulled from the original repository rather than be vendored. The inverse is also true - not only can the backends be removed from the original repository, but so too can any now-unused utilities. Once files were moved or renamed, any references and imports to them needed updating too, across the entire repository history.

In my case, most of the functionality was contained within their own files - there weren't many files which needed to be split up. The biggest annoyance came from the unnecessary changes. For example, in 1 commit I added a feature, the related tests, and added the relevant fixtures into a test file. With the rewrite, the feature and tests were dropped, but because I still need the test fixtures, those remained. That left me with a commit which claims to add a feature, when in reality all it does is add some fixtures. This is fairly simple to work around with some cleanup (more on that later), but it's an annoying and unavoidable side-effect.

#git filter-repo

git-filter-repo is a fantastic tool, and without it this endeavour would have been impossible (or at least far too much effort to bother). git-filter-repo does exactly what it says - it takes a repository and filters it based on a number of rules. Extracting files, renaming files, replacing text. All features I needed, and all features (and more) it has.

#Process

With the plan in place, I started playing around. Because all of this happened locally, it was trivial to retry it multiple times without anyone knowing. Rolling back the rewrite was also simple - just restore the origin, git fetch and hard reset the branch. It meant each iteration took just a few seconds, rather than needing to delete the directory and re-clone.

I cloned the source repository into a separate directory, worked out which files I needed to keep, and how the renames would work. The documentation for git-filter-repo is fairly straightforward, with just a few useful arguments I needed to use:

• --path: Specifies files and folders which will be included in the new repository. Any which aren't matched are removed. This can be specified multiple times, and there are even options for regex and glob matching
• --path-rename: Specifies how to rename a given file or folder, by providing the old and new names. For example, --path-rename django_tasks/backends/database:django_tasks_db renames the django_tasks/backends/database directory to django_tasks_db, and moves everything inside along with it
• --replace-text: Runs a given set of find and replace operations over the text in the repository. This noticeably slowed down execution, but it was worth it to avoid quite so many refactors afterwards. The replacements are specified in a separate file.

Run git filter-repo with the long set of arguments, and it'll give you your new repository. To save you shooting yourself in the foot, it also removes the origin remote, in case you accidentally push it to the wrong place. Initially, I was quite surprised how fast this was to run. Rewriting the history and text of a repository with around 250 commits took under a second, plus the time needed to run the various git gc incantations. I realise git under the hood is just text, and even Python can process a fair bit of that with ease, but still.

#Pruning

Now, sitting in a directory on my computer, was a rewritten repository with just the files I need, and a plausible-looking history to show it wasn't just me who wrote it. However, the history wasn't perfect. As mentioned before, some commits were nonsense and unnecessary. Now is the only time I could easily delete them, and doing so would clean up the history and repository, so it was a no-brainer for me.

What I hadn't quite considered was the number of conflicts I'd end up with. Some of the commits I wanted to replace were half way through the projects lifetime. Where in usual rebases I was processing at most 10 commits, all close together, here I was handling hundreds, all contributing entirely unrelated features, and hoping they didn't depend too much on the specifics I was trying to delete.

In the end, I kept the deletes to a minimum - only deleting those which were completely nonsensical, or introduced features I was about to factor out myself. After a while, it was hard to keep track of the conflicts and what the repository should look like at that point in time. Even days after, I still look back and find the odd commit I could probably have dropped, but I had to move on.

#Manual refactor

With the repository looking as clean as it was going to, I now needed to spend the time making sure the new project worked as expected and met its new purpose. I knew from the start there was always going to be some manual steps to the extraction. Rerunning linters, checking tests, rewriting the README, shuffling around the dependencies and more. These are all fairly simple steps considering what I'd just done, and could be handled as a separate conventional PR. I intentionally did a rebase merge on this PR so the history of the manual tweaks were kept, so it was clearer when something had to be manually changed from the history.

#Closing

All in all, extracting the 2 repositories was a few hours work. From research, testing, running, rerunning and the manual cleanup afterwards. The final deployment was carefully choreographed in the hopes no one noticed and tried to squat the project name before I could get it deployed.

The repositories are now both public for you to see and enjoy, including the manual refactor PR:

• https://github.com/RealOrangeOne/django-tasks-db
• https://github.com/RealOrangeOne/django-tasks-rq

All in all, I'm fairly happy with the outcome. Sure, if I spent more time on each, especially when pruning the unnecessary commits, I could probably have got something even cleaner. But the return on investment on that time would likely never have paid off. All 3 projects exist and are usable, and according to the dependency insights are already being used by eager developers.

Because both of these steps massively rewrote the history, the commit SHAs have also changed, meaning it's almost impossible to attribute changes across the repositories. Because django-tasks is a fairly young project, one I'm hoping will continue for years to come, the slightly messy history shouldn't cause many problems.

#Bonus: Commands

As an extra, and since I'm sure someone is going to ask, these are the commands I used to extract each repository. These commands specifically won't be useful to you, but at least you can see the kinds of cases you need to account for.

#DB Backend

$ git filter-repo --path django_tasks/backends/database --path-rename django_tasks/backends/database:django_tasks_db --path tests/settings.py --path tests/settings_fast.py --path tests/tasks.py --path tests/__init__.py --path tests/db_worker_test_settings.py --path tests/tests/test_database_backend.py --path .github --path .gitignore --path CONTRIBUTING.md --path docker-compose.yml --path justfile --path LICENSE --path manage.py --path pyproject.toml --path django_tasks/py.typed --replace-text ../replace.txt --path-rename django_tasks/py.typed:django_tasks_db/py.typed --path-rename tests/tests/test_database_backend.py:tests/tests.py

django_tasks==>django_tasks_db
django_tasks_db.backends.base==>django_tasks.backends.base
django_tasks_db.base==>django_tasks.base
django_tasks_db.checks==>django_tasks.checks
django_tasks_db.exceptions==>django_tasks.exceptions
django_tasks_db.signals==>django_tasks.signals
django_tasks_db.backends.database==>django_tasks_db
TASK_DEFAULT_PRIORITY==>DEFAULT_TASK_PRIORITY
django-tasks==>django-tasks-db

#RQ Backend

$ git filter-repo --path django_tasks/backends/rq.py --path-rename django_tasks/backends/rq.py:django_tasks_rq/backend.py --path tests/settings.py --path tests/tasks.py --path tests/__init__.py --path tests/tests/test_rq_backend.py --path-rename tests/tests/test_rq_backend.py:tests/tests.py --path .github --path .gitignore --path CONTRIBUTING.md --path justfile --path LICENSE --path manage.py --path pyproject.toml --path django_tasks/py.typed --replace-text ../replace.txt --path-rename django_tasks/py.typed:django_tasks_rq/py.typed

django_tasks==>django_tasks_rq
django_tasks_rq.base==>django_tasks.base
django_tasks_rq.backends.base==>django_tasks.backends.base
django_tasks_rq.checks==>django_tasks.checks
django_tasks_rq.exceptions==>django_tasks.exceptions
django_tasks_rq.signals==>django_tasks.signals
django_tasks_rq.utils==>django_tasks.utils
django-tasks==>django-tasks-rq