Recent content on *scratch*
For a while now we’ve had composefs support in the container-libs. For each OCI layer in an image, we build a read-only EROFS blob that can be verified with fs-verity and that the kernel mounts directly. It works well if you look at a single container, but the moment you start thinking about composefs as the storage for a real container host where many containers are running, we have a…
For the last couple of weeks, I’ve been working on garbage collection for composefs repositories. A composefs repository is a content-addressed object store, and it needs a garbage collector to reclaim space from images that are no longer referenced. The hard part is deciding what “referenced” actually means when an image has no ref in the repository but is mounted somewhere,…
Composefs achieves whole-filesystem integrity verification through image sealing : a single cryptographic digest authenticates an entire filesystem, covering both file contents and metadata (directory structure, permissions, ownership, symlinks, and xattrs). This goes further than fs-verity alone, which can only verify individual file contents, and avoids the fixed-partition requirement of…
I started working on GitChronicler mostly to learn how I could integrate AI into my workflow in a way that would actually spare me doing boring stuff, like writing the git commit message. The tool feeds a patch to a language model via the OpenRouter API and gets back a commit message that reflects what the code actually does — saving the mechanical step of describing changes that are already fully…
It happened a few times in the past that users wonder why they see two /sys/fs/cgroup mounts in their unprivileged container. When working with unprivileged containers in Podman, users often notice two /sys/fs/cgroup mounts if the container is not using a new network namespace. The duplication is not a bug but an intentional consequence of how the kernel handles bind mounts that cross user…
I have been working on a new functionality for the prctl syscall that addresses a common security concern with container runtimes. The /proc/self/exe symlink, which points to the executable of the running process, was the key ingredient in CVE-2019-5736, a vulnerability that allowed a malicious container to overwrite the container runtime binary on the host. The workaround deployed at the time —…
When I started working on crun, I was looking at a faster way to start up and stop containers by improving the OCI runtime, the component in the OCI stack that is responsible for talking to the kernel and setting up the environment where the container runs. Over roughly five years, a combination of kernel patches and userspace fixes reduced the time to start and stop a container from around 160 ms…
A bug report filed against crun a few days ago exposed a deadlock: under certain seccomp profiles, the runtime would hang indefinitely before the container process ever started. The root cause is a subtle sequencing problem between installing a seccomp filter that intercepts a syscall and then immediately using that same syscall to hand off the resulting listener file descriptor to the userspace…
For the last couple of weeks, I’ve been playing on a PoC implementation of a file system for the Linux kernel. The goal is to address a fundamental limitation in how container images are stored: the existing overlay model deduplicates at the layer level, but once you want per-file deduplication — so that identical files across different images share a single copy on disk and in the page…
Seccomp is a kernel feature that restricts what syscalls can be used by a process. The allowed syscalls are described as a BPF program that the kernel evaluates on every syscall entry. While effective, writing and maintaining seccomp profiles in the JSON format expected by OCI runtimes is tedious, and the underlying libseccomp API has surprising constraints — particularly around combining…
One annoying issue with setting a memory limit for a container is that the OOM killer can leave the container in an inconsistent state with only some of its processes terminated. When a cgroup hits its memory limit, the kernel selects a single process to kill based on a badness score, not all the processes in the cgroup. This means that a multi-process container — for example, one running a web…
A couple weekends ago I’ve played with seccomp user notifications and how they can be used in the OCI containers stack. Seccomp user notifications are a Linux kernel feature that lets a privileged monitor process intercept specific syscalls made by a less-privileged container, inspect the arguments, and either emulate the syscall or return an error. This opens up possibilities for safely…
While working on crun, I got surprised by how much time the kernel spent in the copy_mount_options function. A container runtime issues a large number of mount(2) syscalls during startup — bind mounts, proc, sysfs, devtmpfs, and more — many of them with no extra options to pass. It turned out that passing an empty string instead of NULL for the data argument caused the kernel to allocate a full…
CRFS is a Google project that aims at running a container without pre-pulling the image first. The key insight is that in practice a container process only accesses a small fraction of the files in its image, so fetching the entire image before startup wastes both time and disk space. CRFS achieves this through the stargz (Seekable tar.gz) format, which restructures each compressed layer so that…
The giuseppe/crun github project was moved under https://github.com/containers/crun . Moving to the containers organization means the project is no longer a personal experiment but a community-maintained component of the container stack, alongside tools like Podman, Buildah, and fuse-overlayfs. This makes it easier to coordinate changes across the ecosystem and signals that crun is a supported…
I have finally opened some PRs for conmon and libpod that enable resources management for Podman rootless containers on Fedora 30 when using crun. This builds on the cgroups v2 delegation support added to crun earlier: Fedora 30 ships a kernel and systemd new enough to support the unified cgroup hierarchy, so with a single kernel command-line option and a small systemd drop-in, unprivileged users…
cgroups v2 will finally allow unprivileged users to manage a cgroup hierarchy in a safe manner without requiring any additional permission. In the cgroups v1 model, writing to cgroup control files requires root, which means rootless containers cannot enforce memory limits or CPU quotas. The unified cgroups v2 hierarchy introduces a delegation mechanism where systemd can hand ownership of a subtree…
The video of the rootless containers talk from Devconf.cz 2019 is finally available on YouTube. The talk covers how user namespaces, fuse-overlayfs, and slirp4netns come together to allow running containers entirely as an unprivileged user, without any setuid helpers beyond newuidmap and newgidmap, and discusses the remaining challenges around cgroup resource management and overlay storage…
Additional IDs that are allocated to a user through /etc/subuid and /etc/subgid must be considered as permanently allocated and never reused for any other user. The reason is that a setuid binary created inside a user namespace can retain access to any UID that was mapped in that namespace, even after the namespace is destroyed. If the same UID range is later assigned to a different user, that new…
Would be nice to have a way to “fork” the current session and be able to revert all the changes done, without any leftover on the file system. With fuse-overlayfs, a user-space overlay filesystem that unprivileged users can mount, this turns out to be surprisingly straightforward: mount the entire root filesystem as the lower layer of an overlay, point the upper layer at a temporary directory, and…
I was looking for an Emacs mode that could help me to hack on Rust. The built-in rust-mode provides syntax highlighting and basic indentation, but for a language with a complex type system and borrow checker it is useful to have editor integration that can navigate to definitions, show type information, and offer completions. This post covers setting up racer-mode, which drives the racer…
This is the recipe I use to build podman from upstream on Centos 7 and use rootless containers. We need an updated version of the shadow utils as newuidmap and newgidmap are not present on Centos 7. The shadow utils are installed using “make install” which is not the clean way to install packages and it also overwrites the existing binaries, but it is fine on a development system. Podman is…
A couple of weekends ago I’ve played with libslirp and put together slirp-forwarder . The challenge with network namespaces for unprivileged users is that creating TAP or TUN devices requires privileges in the host network namespace. SliRP sidesteps this by emulating a full TCP/IP stack entirely in user space, so the helper process can forward traffic to the outside world using only normal socket…
I’ve cleaned up some C files I was using locally for hacking with user namespaces and uploaded them to a new repository on github: https://github.com/giuseppe/become-root . The tool creates a new user namespace and maps the caller to UID 0 inside it, while also mapping additional UIDs and GIDs from the ranges allocated in /etc/subuid and /etc/subgid . This is the foundation needed for rootless…
The fuse-overlayfs project I was working on in the last weeks was moved under the github.com/containers umbrella. fuse-overlayfs is a user-space implementation of the overlay filesystem that can be mounted without root privileges, which is essential for rootless containers. With Linux 4.18 introducing the ability to mount FUSE filesystems inside user namespaces, this makes overlay-based storage…
Having Buildah running in a user namespace opens the possibility of building container images as a non-root user. I’ve done some work to get Buildah running inside a user container, where it can still create and modify container images without any elevated privileges on the host. This is useful for CI environments and shared systems where granting root or setuid access is not acceptable. There are…
I made a new COPR repository for crun so that it can be easily tested on Fedora without having to build from source. crun is a lightweight OCI container runtime written in C, intended as a faster and lower-overhead alternative to runC. The COPR repository tracks the upstream development branch, making it straightforward to try out new features and report issues before they land in a distribution…
I’ve spent some of the last weeks working on a replacement for runC, the most used/known OCI runtime for running containers. It might not be very well known, but it is a key component for running containers. Every Docker container ultimately runs through runC. The OCI runtime is the thin layer between the container engine and the kernel: it reads a JSON configuration file, creates the necessary…
It is still an ongoing work not ready for production, but the upstream version of OpenShift origin has already an experimental support for running OpenShift Origin using system containers. The “latest” Docker image for origin, node and openvswitch, the 3 components we need, are automatically pushed to docker.io, so we can use these for our test. The rhel7/etcd system container image instead is…
Here are the slides for the Atomic System Containers talk I gave at Devconf.cz 2017. System containers are a way to run infrastructure services — such as etcd and Flannel — outside of Docker, managed directly by runc and systemd, which removes the circular dependency that arises when a container runtime depends on components that must themselves be running inside containers.…
I have been using Facebook for the last years to fill every dead time:waiting for the bus, ads on TV, compiling, etc. The quality of the information coming from Facebook is inferior to any other social network, at least to my experience (it can be I follow/know the wrong people), though the part of the brain that controls procrastination seems addicted to this lower quality information and the…
bubblewrap is a sandboxing tool that allows unprivileged users to run containers. I was recently working on a way to allow unprivileged users to take advantage of bubblewrap to run regular system images that use systemd. To do so, it was necessary to modify bubblewrap to retain a controlled set of Linux capabilities inside the sandbox. Without those capabilities, systemd cannot perform the…
Every programmer at some point gets in touch with the Brainfuck programming language and how surprising is that very few instructions are needed to have a Turing complete language, 6 is the case of Brainfuck (plus other 2 for I/O operations). The language operates on a tape of memory cells using only increment, decrement, pointer movement, and conditional loop instructions. Writing a GCC frontend…
git rebase is one of my favorite git commands. It allows to update a set of local patches against another git branch and also to rework, through the -i flag, some previous patches. A lesser-known capability is the –exec flag, which runs an arbitrary shell command after each patch is applied during the rebase. Combined with the -X theirs merge strategy to silently resolve conflicts, this…
The main reason behind system containers was the inability to run Flannel in a Docker container as Flannel is required by Docker itself. CoreOS solved this chicken and egg problem by using another instance of Docker (called early-docker) that is used to setup only Etcd and Flannel. Atomic system containers take a different approach: instead of a second Docker daemon, they are managed directly by…
rpm-ostree, used together with OStree, is a powerful tool to generate immutable images for .rpm based systems, so why not use it for generating Docker images as well? rpm-ostree already has support for composing a container tree, but the steps to go from an OStree commit to a ready-to-run Docker image involve several manual operations. ostree-docker-builder is a small tool that automates this…
Coming as a surprise, this year we have got 4 students to work full-time during the summer on wget. More than all the students who have ever worked for wget before during a Summer of Code! The projects span a broad range of improvements: HSTS and FTPS security hardening, TCP Fast Open and conditional GET for performance, HTTP/2 support on top of nghttp2, and an expanded FTP test suite. Each…
This tutorial shows how to create a QCOW2 image that can be directly imported via virt-install to test out Fedora 22 Atomic starting from a custom OStree repo. The process involves composing an OStree repository from a Fedora Atomic tree definition, serving it over HTTP so the installer can reach it, then driving an unattended installation through a kickstart file. This gives you full control over…
These are the steps to setup the current website in a Docker container. The setup uses docker-compose to declaratively describe a two-container application: a MySQL 5.5 database and a WordPress frontend. Docker links tie them together so that the web container can reach the database by hostname, without needing to hard-code any IP addresses or manage networking manually. A single docker-compose up…
Software engineer at Red Hat, based in Rome, Italy. Most of my work in the last years was on container runtimes, especially Podman and the crun OCI runtime. A more detailed list of my contributions is on GitHub . To contact me, you can write an email at giuseppe@ followed by the same domain here. My PGP key fingerprint is AC40 4C1C 0BF7 35C6 3FF4 D562 263D 6DF2 E163 E1EA ( keys.openpgp.org ).…