I've been trying to think of ways to speed up image builds for the local package layering use case. The challenge is making big RPM transactions in image builds more incremental, so that the time cost of adding an additional package to a 2,391-package bootc system is closer to 1 than 2,391 + 1.
I think an important part of that is going to be caching uncompressed RPMs in a deduplicated way, similar to how rpm-ostree imports RPMs to OSTree commits.
I'm imagining a variant of RPM CoW where the transcode step is unpacking the RPM payload into a composefs store, and replacing the payload inside the RPM with an EROFS/composefs metadata image that references files in the composefs store. The install step would then reflink files from the composefs store to the installroot, rather than from the extents-transcoded RPMs.
The main advantage over extents-based RPM CoW is that all transcoded RPMs with the same backing store would be automatically duplicated at the file level. This is nice for keeping old versions of RPMs around in case you need to roll back. (yes, you could roll back the entire image, but it's also good to be able to roll back to a state you can efficiently edit/rebuild).
Garbage collection would be trickier than with extents-based CoW. Deleting an RPM would not free the space in the composefs store. So rpm2composefs would make more sense when managed by a DNF plugin or image building system that can be responsible for GC.
This would depend on additions to the plugin API: #2057.
Um. That's is flying with so many technologies and whatnot that I've just barely even heard of that it's difficult to follow. And referring to CoW in rpm context, at this point only manages to raise my hackles. That's not a good place to start at 😅
Lets start from the basics. The use-case seems to be optimizing repeated similar/identical transactions, nevermind the implementation details. On systems that are used for image building. And so a cache as an optimization technique makes sense. Right? If so, that is something we can discuss.
1 reply
Did not mean to raise any hackles :) This is just an idea I'm toying with, discussion of the additions to the plugin API (#2057.) should probably stay focused on the more mature RPM CoW proposal.
The use-case seems to be optimizing repeated similar/identical transactions, nevermind the implementation details. On systems that are used for image building. And so a cache as an optimization technique makes sense. Right?
Yes, exactly.
Yep, it's primarily about optimizing for disk I/O (speed) in the environments where you run many transactions with largely the same content, on the same (file)system. Image builds are the most prominent example. Here, RPMs could be thought of as basically just recipes for composing and merging filesystem trees (packages) from files stored in a single (ideally content-addressable) location on disk, rather than being the delivery medium itself.
I can see two ways to approach this:
- Natively in RPM, at the payload handling level, using a standard mechanism like reflinks or similar
- Externally to RPM, by decoupling the payload handling and exposing an API for it (in the spirit of API improvement to accommodate for RPM CoW (PR#1470) #2057)
5 replies
So the idea is that the payload would not be the actual payload, but a list of references to an external source of artifacts used to produce the same state that the installation of the package would otherwise produce..?
(Maybe decouple this if possible from the specifics of that artifact source, because one could imagine that source of artifacts taking many potential forms or evolving over time..)
- Externally to RPM, by decoupling the payload handling and exposing an API for it (in the spirit of API improvement to accommodate for RPM CoW (PR#1470) #2057)
That's what I was thinking.
So the idea is that the payload would not be the actual payload, but a list of references to an external source of artifacts used to produce the same state that the installation of the package would otherwise produce..?
Yes, unlike RPMCoW, the actual payload contents would not be stored within the RPM files themselves. So the applications here would be for caching RPMs locally rather than sending them around. And this would probably not fit into RPM's security model... unless the backing store could guarantee the cryptographic integrity of the files, like composefs+fs-verity.
Come to think of it though, you could still use composefs as a content-addressed store for file-level deduplication using the original RPMCoW's transcoded RPM format, which does include the payload file contents. While transcoding the RPM, rpm2extents could check whether each file is in the composefs store. If it is, FICLONERANGE it from the composefs store into the transcoded RPM instead of copying it from the original extracted RPM. If it's not, add it to the composefs store.
Somewhere along the previous CoW discussions I came up with the vague idea of a hash-addressed storage (just unpacked individual files on disk, think .git/objects style directory somewhere, the hash automatically dedupes) that you could instruct rpm to use (populate as needed, copy/reflink as appropriate). Rpm can verify those files during the unpacking as it normally would, so integrity is preserved.
In such a mode, rpm could basically operate the transaction on packages stripped off their payloads, except for the package verification at the start of the transaction which verifies the payload as well. The payload stream contains the cpio headers for v4 packages and index numbers for v6 packages, so you can't just toss the uncompressed files into a digest and come up with the same value. Generating those cpio headers on the fly would be somewhat painful, but for v6 packages it would be fairly easy (I think), and there you have a means to verify the package in its entirety, as if coming from a good ol' rpm file.
This kind of operation might be close enough to rpm's core business that it wouldn't have to go into a plugin. Which has both pros and cons.
Somewhere along the previous CoW discussions I came up with the vague idea of a hash-addressed storage (just unpacked individual files on disk, think .git/objects style directory somewhere, the hash automatically dedupes
That's what https://github.com/composefs/composefs-rs is doing (and ostree has been doing something like this too) but composefs is a lot better because it integrates with fsverity to give on-disk integrity.
One thing that would be helpful - I think we could add the fsverity digest (one of or both fsverity-sha256-12 and fsverity-sha512-12) for all (or just >64b files like composefs does) to the header as an addition to the existing classic sha256 etc - or even just eventually hard cutover to fsverity, because it's just better to have the kernel support verifying it.
That would allow us to always efficiently look up an object by digest in the composefs store.
I've been trying to think of ways to speed up image builds for the local package layering use case. The challenge is making big RPM transactions in image builds more incremental, so that the time cost of adding an additional package to a 2,391-package bootc system is closer to 1 than 2,391 + 1
More efficient algorithms are always good, but it might be worth trying another approach first, which is to make rpm work faster and see if that is sufficient. This week I made rpm the second major test case for LumoSQL and it does indeed appear to make rpm work faster for me for the operations I tried. You compile up the LumoSQL library choosing LMDBv1.0 as the backend, change its soname with patchelf, and point LD_LIBRARY_PATH at the new libsqlite3. rpm doesn't know the difference (the file format is incompatible so you need to export/import to the new format to do performance testing.)
LumoSQL also gives you encrypted rpm and checksummed rpm if you wish, but they won't help it go faster. I can share my recipe if you like, although it isn't polished.
--
Dan Shearer
dan@shearer.org / https://shearer.org
0 replies