Setting up Gateway API support in K3s

With Ingresses slowly on their way out, I decided to take a look at implementing support for the Gateway API in the K3s cluster.

The Gateway API implementation had to work similar to my current Traefik Ingresscontroller setup:

• A single external Virtual IP (VIP) that can be used for all internal DNS records in Unbound running in OPNsense
• Support for TLS certificate provisioning using my internal CA and cert-manager

I researched a few alternatives that were conformant and noticed that Calico was listed. I found their documentation on it:

• Calico Ingress Gateway
• Creating Ingress Gateway
• Customizing Ingress Gateway

Turns out that its Envoy Gateway under the hood. Seemed promising and the Tigera Operator does all the heavy lifting. The prerequisites are covered by my existing setup:

• Tigera Operator was used to bootstrap Calico in K3s
• I’m already using MetalLB to provision services of type LoadBalancer

Having cleared all blockers, I started on the implementation.

MetalLB and GatewayClass

MetalLB needed a new IPAddressPool and L2Advertisement to announce the external IP of the gateways using the new GatewayClass.

I chose 10.0.3.204 to be the external IP address. The pool will only hold that single IP:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: tigera-gateway
  namespace: metallb-system
spec:
  addresses:
  - 10.0.3.204-10.0.3.204

Advertised using Layer2 mode:

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: tigera-gateway
  namespace: metallb-system
spec:
  ipAddressPools:
  - tigera-gateway

The Ingress Gateway docs also state that a default deny policy must exclude the namespace tigera-gateway in its namespaceSelector:

If you are using a global default deny policy, allow traffic through the gateway by adding the tigera-gateway namespace to the list of excluded namespaces in the namespaceSelector field.

I adapted all existing GlobalNetworkPolicy resources to exclude it (this is just one of the polices, the diff is the same for the other policies):

--- a/calico/default_deny_all_netpol.yaml
+++ b/calico/default_deny_all_netpol.yaml
@@ -3,7 +3,7 @@ kind: GlobalNetworkPolicy
 metadata:
   name: default-deny
 spec:
-  namespaceSelector: kubernetes.io/metadata.name not in {"kube-system", "kube-public", "calico-system", "calico-apiserver"}
+  namespaceSelector: kubernetes.io/metadata.name not in {"kube-system", "kube-public", "calico-system", "calico-apiserver", "tigera-gateway"}
   types:
   - Ingress
   - Egress

I then created the GatewayAPI resource itself. The important part is adding the metallb.io/address-pool annotation on the Service of the Gateway to make it use the new MetalLB IPAddressPool.

apiVersion: operator.tigera.io/v1
kind: GatewayAPI
metadata:
  name: default
spec:
  gatewayClasses:
    - name: tigera-gateway-class
      gatewayService:
        metadata:
          annotations:
            # Ensures the service is using the correct
            # MetalLB address-pool for its external
            # IP address.
            metallb.io/address-pool: tigera-gateway

The namespace tigera-gateway is created by the Tigera Operator when the GatewayAPI resource is created.

All the resources were applied using Argo, but something was not working:

k get gatewayclasses.gateway.networking.k8s.io
NAME                   CONTROLLER                                      ACCEPTED   AGE
tigera-gateway-class   gateway.envoyproxy.io/gatewayclass-controller   Unknown    71s

k get pods -n tigera-gateway
NAME                                            READY   STATUS              RESTARTS   AGE
envoy-gateway-544ff899b4-dj47p                  0/1     ContainerCreating   0          3m34s
tigera-gateway-api-gateway-helm-certgen-rvj67   0/1     Error               0          64s
tigera-gateway-api-gateway-helm-certgen-xmrc9   1/1     Running             0          21s

The *-certgen-* pods had some issues and crashed for a while. I tried deleting the entire GatewayAPI resource and resync in Argo. I had a few minutes of the following errors in the logs of the envoy-gateway pod before the issue resolved itself:

{"level":"info",...,"msg":"Failed to query deployment","reason":"Deployment.apps \"envoy-gateway\" not found"}
{"level":"info",...,"msg":"Failed to query deployment","reason":"Deployment.apps \"envoy-gateway\" not found"}
{"level":"info",...,"msg":"Failed to query deployment","reason":"Deployment.apps \"envoy-gateway\" not found"}

Once it resolved I had a working GatewayClass:

k get gatewayclasses.gateway.networking.k8s.io
NAME                   CONTROLLER                                      ACCEPTED   AGE
tigera-gateway-class   gateway.envoyproxy.io/gatewayclass-controller   True       9m37s

Gateway and HTTPRoute

For testing I used *.gw.homelab.fredrickb.com as the subdomain instead of the existing *.homelab.fredrickb.com subdomain. The final version of resources using the correct subdomain is shown in Cleanup of Gateway, HTTPRoute and OPNsense configuration.

Gateway for testing was created (docs):

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: test-gateway
  namespace: calico-system
spec:
  gatewayClassName: tigera-gateway-class
  listeners:
  - protocol: HTTP
    port: 80
    name: http
    allowedRoutes:
      namespaces:
        from: Same

The Service for the Envoy proxy was bootstrapped automatically:

k get svc -n tigera-gateway | grep LoadBalancer
envoy-calico-system-test-gateway-9a2df979   LoadBalancer   10.43.142.115   10.0.3.204    80:32101/TCP                                       2m17s

Eventually the Gateway was programmed with the expected IP address:

k get gateway -A
NAMESPACE       NAME           CLASS                  ADDRESS      PROGRAMMED   AGE
calico-system   test-gateway   tigera-gateway-class   10.0.3.204   True         3m14s

I verified it was responding on the expected port:

nmap -Pn -p 80 10.0.3.204
...
PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.02 seconds

Then I proceeded to create the HTTPRoute for testing:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: whisker
  namespace: calico-system
spec:
  parentRefs:
  - name: test-gateway
  hostnames:
  - whisker.gw.homelab.fredrickb.com
  rules:
  - backendRefs:
    - name: whisker
      port: 8081

Followed up by adding an Unbound alias for the domain to test it:

I tested it after applying the changes, but it did not work:

curl whisker.gw.homelab.fredrickb.com/flow-logs
upstream connect error or disconnect/reset before headers. reset reason: connection timeout

The logs of the Envoy proxy itself had no additional info. When I checked Whisker (using the domain of the existing ingress) it had caught the error:

The issue was the existing NetworkPolicy not allowing access from the Envoy proxy to the Whisker service. I made the following changes:

--- a/calico/whisker_netpol.yaml
+++ b/calico/whisker_netpol.yaml
@@ -11,6 +11,12 @@ spec:
       podSelector:
         matchLabels:
           app.kubernetes.io/name: traefik
+    - namespaceSelector:
+        matchLabels:
+          kubernetes.io/metadata.name: tigera-gateway
+      podSelector:
+        matchLabels:
+          app.kubernetes.io/name: envoy

Then it worked:

curl whisker.gw.homelab.fredrickb.com/flow-logs
<!doctype html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <link rel="icon" href="public/favicon.ico?v=2" />
        <title>Calico Whisker</title>
    <script defer src="/static/js/lib-react.aebd6017.js"></script><script defer src="/static/js/lib-router.e1df1bfb.js"></script><script defer src="/static/js/7.b44bd4e6.js"></script><script defer src="/static/js/index.412f5017.js"></script><link href="/static/css/index.934858bb.css" rel="stylesheet"></head>
    <body>
        <div id="root"></div>
    </body>
</html>

cert-manager

With ordinary HTTP working it was time to fix certificate provisioning and TLS for gateway traffic. Enabling Gateway API support is detailed in the cert-manager docs. I updated the Argo Application for the cert-manager Helm release to include it:

--- a/cert-manager/cert_manager_app.yaml
+++ b/cert-manager/cert_manager_app.yaml
@@ -11,8 +11,10 @@ spec:
     targetRevision: 1.19.4
     helm:
       releaseName: cert-manager
-      values: |
+      valuesObject:
         installCRDs: true
+        config:
+          enableGatewayAPI: true
   destination:
     server: "https://kubernetes.default.svc"
     namespace: cert-manager

I then changed the Gateway accordingly to make it use the correct StepClusterIssuer to provision certificates using my internal CA. I didn’t want to make gateways for each namespace so I opted for a single gateway with a wildcard hostname, matching all subdomains of *.gw.homelab.fredrickb.com. It also routes to all namespaces:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: calico-gateway
  annotations:
    cert-manager.io/issuer-group: certmanager.step.sm
    cert-manager.io/issuer-kind: StepClusterIssuer
    cert-manager.io/issuer: step-issuer
spec:
  gatewayClassName: tigera-gateway-class
  listeners:
  - protocol: HTTPS
    hostname: "*.gw.homelab.fredrickb.com"
    port: 443
    name: https
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - name: homelab-fredrickb-com-tls

The certificate was then provisioned:

k -n calico-system get certificate
NAME                        READY   SECRET                      AGE
homelab-fredrickb-com-tls   True    homelab-fredrickb-com-tls   84s
...

And now TLS was working:

curl https://whisker.gw.homelab.fredrickb.com/flow-logs
<!doctype html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <link rel="icon" href="public/favicon.ico?v=2" />
        <title>Calico Whisker</title>
    <script defer src="/static/js/lib-react.aebd6017.js"></script><script defer src="/static/js/lib-router.e1df1bfb.js"></script><script defer src="/static/js/7.b44bd4e6.js"></script><script defer src="/static/js/index.412f5017.js"></script><link href="/static/css/index.934858bb.css" rel="stylesheet"></head>
    <body>
        <div id="root"></div>
    </body>
</html>

Cleanup of Gateway, HTTPRoute and OPNsense configuration

I used the subdomain *.gw.homelab.fredrickb.com for testing gateway traffic. I wanted this to be *.homelab.fredrickb.com to match what I’m currently using for ingresses. Which IP address a subdomain is resolved to can then be set just by moving the alias on the host override in Unbound.

I started by deleting the existing resources used for testing:

k -n calico-system delete httproute whisker
httproute.gateway.networking.k8s.io "whisker" deleted from calico-system namespace
k -n calico-system delete gateway test-gateway
gateway.gateway.networking.k8s.io "test-gateway" deleted from calico-system namespace

Removed the section I added to the NetworkPolicy earlier for testing:

--- a/calico/whisker_netpol.yaml
+++ b/calico/whisker_netpol.yaml
@@ -11,12 +11,6 @@ spec:
       podSelector:
         matchLabels:
           app.kubernetes.io/name: traefik
-    - namespaceSelector:
-        matchLabels:
-          kubernetes.io/metadata.name: tigera-gateway
-      podSelector:
-        matchLabels:
-          app.kubernetes.io/name: envoy

Created a new NetworkPolicy to allow access from Envoy proxy to Whisker:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-tigera-gateway-api-to-whisker
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: tigera-gateway
      podSelector:
        matchLabels:
          app.kubernetes.io/name: envoy
    ports:
    - port: 8081
      protocol: TCP
  podSelector:
    matchLabels:
      app.kubernetes.io/name: whisker
  policyTypes:
  - Ingress

Changed the Gateway to have the correct hostname:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: calico-gateway
  annotations:
    cert-manager.io/issuer-group: certmanager.step.sm
    cert-manager.io/issuer-kind: StepClusterIssuer
    cert-manager.io/issuer: step-issuer
spec:
  gatewayClassName: tigera-gateway-class
  listeners:
  - protocol: HTTPS
    hostname: "*.homelab.fredrickb.com"
    port: 443
    name: https
    allowedRoutes:
      namespaces:
        from: All
    tls:
      mode: Terminate
      certificateRefs:
      - name: homelab-fredrickb-com-tls

Changed the HTTPRoute to have the correct hostnames:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: whisker
spec:
  parentRefs:
  - name: calico-gateway
  hostnames:
  - whisker.homelab.fredrickb.com
  rules:
  - backendRefs:
    - name: whisker
      port: 8081

Removed the Whisker alias from the Traefik host override in Unbound:

Added the Whisker alias to the Gateway host override in Unbound:

Verified that the DNS changes were propagated:

dig @10.0.0.1 whisker.homelab.fredrickb.com
...
whisker.homelab.fredrickb.com. 3600 IN  A       10.0.3.204
...

Ensured the HTTPRoute was reachable using the new domain:

curl https://whisker.homelab.fredrickb.com/flow-logs
<!doctype html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <link rel="icon" href="public/favicon.ico?v=2" />
        <title>Calico Whisker</title>
    <script defer src="/static/js/lib-react.aebd6017.js"></script><script defer src="/static/js/lib-router.e1df1bfb.js"></script><script defer src="/static/js/7.b44bd4e6.js"></script><script defer src="/static/js/index.412f5017.js"></script><link href="/static/css/index.934858bb.css" rel="stylesheet"></head>
    <body>
        <div id="root"></div>
    </body>
</html>

Removed the Ingress and NetworkPolicy resources for Whisker since it will no longer be exposed using the Traefik ingresscontroller:

--- a/calico/whisker_ingress.yaml
+++ /dev/null
@@ -1,24 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  annotations:
-    cert-manager.io/issuer: step-issuer
-    cert-manager.io/issuer-group: certmanager.step.sm
-    cert-manager.io/issuer-kind: StepClusterIssuer
-  name: whisker
-spec:
-  rules:
-  - host: whisker.homelab.fredrickb.com
-    http:
-      paths:
-      - backend:
-          service:
-            name: whisker
-            port:
-              number: 8081
-        path: /
-        pathType: Prefix
-  tls:
-  - hosts:
-    - whisker.homelab.fredrickb.com
-    secretName: whisker-tls-cert

--- a/calico/whisker_netpol.yaml
+++ /dev/null
@@ -1,21 +0,0 @@
-apiVersion: networking.k8s.io/v1
-kind: NetworkPolicy
-metadata:
-  name: allow-traefik-to-whisker
-spec:
-  ingress:
-  - from:
-    - namespaceSelector:
-        matchLabels:
-          kubernetes.io/metadata.name: kube-system
-      podSelector:
-        matchLabels:
-          app.kubernetes.io/name: traefik
-    ports:
-    - port: 8081
-      protocol: TCP
-  podSelector:
-    matchLabels:
-      app.kubernetes.io/name: whisker
-  policyTypes:
-  - Ingress

Fixing remote access

When testing remote access to Whisker it didn’t work. I had forgotten to add a firewall rule to allow access to the VIP of the Gateway from the VPN VLAN.

I created a new firewall alias in OPNsense:

I then added the firewall alias to a new firewall rule:

End result:

Remote access worked afterwards.

Conclusion

I now have a foundation I can build on to start converting my Ingress resources to HTTPRoute while still supporting both APIs. I might experiment with other implementations of the Gateway API in the future, but for now this works.