Connor's Blog

Creating a Private DNF Repo With S3

I was recently talking with a friend about setting up a Rocky Linux repository mirror in S3. He managed to find a plugin for yum capable of grabbing instance IAM credentials from the metadata API and building the necessary headers to authenticate with S3. Unfortunately, he couldn’t get it to work.

I took some interest and decided to have a look myself. My suspicion for the plugin not working is that yum is essentially deprecated and now dnf is the default package manager on Rocky/Red Hat based distributions. dnf supports plugins too although has a slightly different plugin API to yum, so I attempted to rewrite it.

The dnf plugin API supports setting request headers for each repository, so that looked like a good sign. But long story short, with dnf plugins it is no longer possible to intercept requests using the same trick as the original plugin used. It did a clever bit of moneypatching to replace the URLGrabber object. The reason requests need to be intercepted is that building the SigV4 signature sent in each request requires knowledge of the request path and query parameters.

With the idea of a plugin out of the window, I had a look into proxying requests from the package manager instead. Fortunately I didn’t need to write my own proxy because Amazon had already built one called aws-sigv4-proxy.

Here’s the steps to set it up with dnf:

  1. Start it up
$ docker run --rm -d -p 8080:8080 public.ecr.aws/aws-observability/aws-sigv4-proxy
  1. Create a new .repo file in /etc/yum.repos.d
[<REPO NAME>]
name = <REPO NAME>
baseurl = http://s3.<BUCKET REGION>.amazonaws.com/<BUCKET NAME>/<REPO NAME>
proxy = http://localhost:8080

where <REPO NAME> would be the name of a DNF repository, like baseos or appstream. Should be a directory in your bucket. Note that although the baseurl is a HTTP URL, the proxy makes a HTTPS connection to S3.

  1. Enjoy

  2. Consider enabling GPG checking, for example

[<REPO NAME>]
...
+ gpgcheck = 1
+ gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-Rocky-9

You can find an issue tracking the plugin’s support for dnf and the comments documenting my findings originally here.