Not-forking ↗ exists to solve a difficult problem presented in the LumoSQL project: how could we graft two very active source trees together in an ongoing way to produce a third, independent tree? I designed and tested the first phase of Not-forking, and Claudio Calvelli did most of the coding. These days I’m the maintainer, and it turns out Not-forking is a reproducibility tool which addresses the computer science problem of developer intent.

More specificially, Not-forking avoids project-level forking by automating change management in ways that version control systems ↗ such as Fossil ↗, Git ↗, or GitHub ↗ cannot.

Not-forking:

  • lets you integrate non-diffable codebases, a bit like patch/sed/diff/cp/mv rolled into one.
  • is a machine-readable file format and tool. It answers the question: What is the minimum difference between multiple source trees, and how can this difference be applied as versions change over time?
  • avoids duplicating source code. When one project is within another project, and the projects are external to each other, there is often pressure to fork the inner project. Not-forking avoids that.
  • helps address the problem of reproducibility. By giving much better control over the input source trees, it is more likely that the output binaries are the same each time.

The full documentation ↗ goes into much more detail than this overview.

This following diagram shows the simplest case:

Some questions immediately arise:

  • Should you import Upstream into your source code management system?
  • If Upstream makes modifications, how can you pull those modifications into Combined Project safely?
  • If Combined Project has changed files in Upstream, how can you merge them safely?

Not-forking also addresses more complicated scenarios:

And even more complex cases:

Why Not Just Use Git/Fossil/Other VCS?

Git rebase cannot solve the Not-forking problem space. Neither can Git submodules. Nor Fossil’s merge, nor the quilt approach to combining patches.

A VCS cannot address the Not-forking class of problems because the decisions required are typically made by humans doing a port or reimplementation where multiple upstreams need to be combined. A patch stream can’t describe what needs to be done, so automating this requires a tangle of fragile one-off code. Not-forking makes it possible to write a build system without these code tangles.

Examples of the sorts of actions Not-forking can take:

  • check for new versions of all upstreams, doing comparisons of the human-readable release numbers/letters rather than repo checkins or tags, where human-readable version numbers vary widely in their construction
  • replace foo.c with bar.c in all cases (perhaps because we want to replace a library that has an identical API with a safer implementation
  • apply this patch to main.c of Upstream 0, but only in the case where we are also pulling in upstream1.c, but not if we are also using upstream2.c
  • apply these non-patch changes to Upstream 0 main.c in the style of sed rather than patch, making it possible to merge trees that a VCS says are unmergable
  • build with upstream1.c version 2, and upstream3.c version 3, both of which are ported to upstream 0’s main.c version 5
  • track changes in all upstreams, which may use arbitrary release mechanisms (Git, tarball, Fossil, other)
  • cache all versions of all upstreams, so that a build system can step through a large matrix of versions of code quickly, perhaps for test/benchmark

Where to next for Not-forking?

The project evolved from a design to solve one specific class of problems into a more general reproducibility tool where the requirement is to construct reliable binary artefacts from multiple dynamic input trees. This is the sort of thing that gives the reproducible-builds ↗ community headaches and there is a lot to explore. The not-forking recipe mechanism encapsulates transformations that are not expressible in any standard way, but which also look familiar to an experienced eye because they are the sort of thing that is re-invented constantly in tree-specific, one-off hacks. The documentation always was careful and quite formal (Claudio Calvelli has the excellent habit of writing the documentation first) and suggests what a more comprehensive normative form might look like. The problem space could be explored thoroughly and a standard for recipes be developed to unambiguously and safely make these transformations in a way that generalises across trees, time and technologies. If you would like to work on practical reproducibility

From a theoretical point of view, Not-forking has brought up an entire class of problem related to developer intent. There is relatively little literature on this, which is why I rescued two of Margaret Hamilton’s print articles ↗. As that article says:

Vertical intent is the question that very few focus on, that of whether what we say in code is faithfully transmitted by the toolchain down to the running silicon. This is Ken Thompson’s [Reflections on Trusting Trust](https: //dl.acm.org/doi/10.1145/358198.358210), and the [xz supply chain incident] (https://en.wikipedia.org/wiki/XZ_Utils_backdoor) ↗, and is what I was looking for.

Feel free to get in touch if any of the three things detailed above interest you:

  1. a better not-forking tool, able to be applied in production to other difficult open source tree management tasks
  2. standard description of not-forking recipes, which will require an exhaustive exploration of the problem space and at least one paper on what is being solved
  3. publishing an exploratory computer science paper around the topic of Vertical Intent

… and if anyone wants to contribute a better name that encapsulates what not-forking does in the title, please be our guest!