Lightweight container building blocks

I've been playing the last few weeks with Podman as a container manager. (I say "manager" because, as I understand it, Podman is mainly in charge of resolving and fetching the container layers, configuring the networking, and then delegating to an actual container "runtime" such as crun or runc.)

However, this article is not about Podman, but it serves to know where I'm coming from.

I've played with it both in root-less (as a normal user) and root-full (as root user) modes, enough to say what are my main issues with Podman:

• it strives to be a drop-in replacement to Docker;

  thus although it reimplements everything from scratch, at least the user-facing parts (CLI, API, etc.) and the Docker-legacy parts (mainly container images) are exactly the same;

• it strives to be excessively enterprise-friendly, by supporting a plethora of infrastructure related components, from storage to networking;

  and unfortunately it shows, because nothing feels polished enough to feel production ready (at least to me); (in fact what is "production ready" these days?)

(Many, many, years ago, just before Docker was released, I've played with LXC and VServer, and I still find their proposition somewhat saner than Docker's...)

Fortunately, Podman was only the first step in my experiments, and I've since sought alternative tools that could help in building a more lightweight, customized solution.

In this article, I will briefly list these interesting projects, and my initial thoughts about them.

Note I've not extensively used any of these, and for some I've only read the documentation or GitHub readme! But will keep an eye on their future developments.

Container image tools

• github.com/containers/skopeo

  Part of the Podman suite, its main purpose is to interact with OCI (i.e. Docker compatible) images and repositories.

• github.com/containers/buildah

  Also part of the Podman suite, it's perhaps a saner replacement to the Dockerfile, providing one more flexibility than the original.

• github.com/opencontainers/umoci

  An interesting tool to interact, export and import, OCI compliant images. Thus enabling one to escape the whole OCI compliant environment, but still build upon existing container registries.

• github.com/ostreedev/ostree

  Although not currently used as a container technology, perhaps except for Flatpack, OSTree can be seen as "the git for file-systems".

  It can be a good base for building some alternative container storage and composition tools, especially in cooperation with skopeo and umoci.

Container execution tools

• github.com/containers/crun

  Another building block part of the Podman ecosystem, this enables one to execute OCI compliant containers, after one has prepared everything that is necessary.

  If one does need the functionality of Podman, but with a different frontend, crun can be called directly.

• github.com/containers/youki

  This is a Rust reimplementation of crun, also part of the Podman ecosystem.

• github.com/containers/bubblewrap

  This is an even lightweight variant of crun, although not an OCI compliant runtime, that enables one to easily run something in a Linux namespace, especially with mount and user separation. (Namespaces are the "magic secret sauce" behind all Linux container technology...)

  It is currently being used by Flatpack for running self-contained applications as containers.

• github.com/google/minijail

  Released just a few days ago, minijail is very similar to bubblewrap, however my impression is that it focuses more on security than just virtualization.

  It seems it is used for "sandboxing and containment in ChromeOS and Android" (quoting from the readme).

  (But being a Google-sponsored project, who knows when Google will decide to drop development and support...)

Although full-virtualization is exactly what the container technologies have tried to provide alternatives to, many businesses use MicroVM virtualization solutions in place of containers. For example Fly.io, AWS Lambda, AWS Fargate, CloudFlare WebWorkers, just to name the most popular ones.

(Long term, I would bet that much of the current container technologies and runtimes will be replaced with MicroVM alternatives.)

MicroVM runtimes

• github.com/firecracker-microvm/firecracker

  AWS developed, it's the virtualization behind AWS Lambda, AWS Fargate, and others such as Fly.io.

  It boots a minimal Linux kernel and initramfs in a couple of milliseconds, thus comparable with many container runtimes in terms of latency.

• QEMU -- 'microvm' virtual platform

  QEMU's take on the whole MicroVM movement.

• github.com/cloud-hypervisor/cloud-hypervisor

  Perhaps not as lightweight a proper MicroVM, it is perhaps a good alternative when someone needs something in between a full-blown QEMU and a minimalist Firecracker.

PID-1 init alternatives

I won't enter into the details here, but if one runs in a container anything that forks or executes other processes, then one most likely needs a proper init solution, although a minimal one tailored for containers.

• github.com/krallin/tini
• github.com/openSUSE/catatonit
• github.com/Yelp/dumb-init

I'll keep updating this article as I find new and interesting projects pertaining to the subject.

Also, feel welcome to write me an email if you feel something is missing from this list. (Please note I'm searching for lightweight building blocks, not fully-fledged solutions.)