AWS style virtual-host buckets for Rook Ceph on OpenShift

Background#

AWS deprecated path-style access for S3 buckets for virtual-host accessing method. Ever since, it is generally advisable to use virtual-host style accessing in applications.

Path-style access model#

https://s3.amazonaws.com/jbarr-public/images/ritchie_and_thompson_pdp11.jpeg
https://s3.amazonaws.com/jsb-public/classic_amazon_door_desk.png

Virtual-host style access model#

https://jbarr-public.s3.amazonaws.com/images/ritchie_and_thompson_pdp11.jpeg
https://jsb-public.s3.amazonaws.com/classic_amazon_door_desk.png

Rook’s CephObjectStore CR uses path-style addressing by default. In this blog post we’ll explore how to enable virtual-host style addressing for Ceph RGW.

Steps#

1. First, we need to ensure a wildcard certificate exists for the S3 endpoint.

   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: public-rgw-internal-nanibot-net-cert
     namespace: rook-ceph
   spec:
     secretName: public-rgw-internal-nanibot-net-cert-secret
     duration: 2160h
     renewBefore: 720h
     privateKey:
       algorithm: RSA
       encoding: PKCS1
       size: 2048
       rotationPolicy: Always
     dnsNames:
       - "*.ceph-objectstore.apps.openshift.internal.nanibot.net"
     issuerRef:
       name: nanibot-net-issuer
       kind: ClusterIssuer
   

2. Modify the default IngressController to allow wildcard routes by setting spec.routeAdmission.wildcardPolicy.

   apiVersion: operator.openshift.io/v1
   kind: IngressController
   metadata:
     name: default
     namespace: openshift-ingress-operator
   spec:
     ...
     routeAdmission:
       wildcardPolicy: WildcardsAllowed
     ...
   

3. Set the spec.hosting.dnsNames field for the CephObjectStore to include the wildcard addressable domain name (Note: Don’t include the * character).

   apiVersion: ceph.rook.io/v1
   kind: CephObjectStore
   metadata:
     name: ceph-objectstore
     namespace: rook-ceph
   spec:
     ...
     hosting:
       dnsNames:
         - ceph-objectstore.apps.openshift.internal.nanibot.net
   

4. Finally, create an ingress for the S3 endpoint. OpenShift’s Route Controller Manager will convert the ingress to an appropriate route.

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: ceph-objectstore-ingress
     namespace: rook-ceph
     annotations:
       route.openshift.io/termination: "reencrypt"
       route.openshift.io/destination-ca-certificate-secret: pki-production-ca
   spec:
     ingressClassName: openshift-default
     tls:
     - secretName: public-rgw-internal-nanibot-net-cert-secret
       hosts:
         - '*.ceph-objectstore.apps.openshift.internal.nanibot.net'
     rules:
     - host: '*.ceph-objectstore.apps.openshift.internal.nanibot.net'
       http:
         paths:
         - path: '/'
           pathType: Prefix
           backend:
             service:
               name: rook-ceph-rgw-ceph-objectstore
               port:
                 name: https
   

Note: The above will generate a route of type “reencrypt”. The annotation “route.openshift.io/destination-ca-certificate-secret” points to the secret containing the CA certificate of the CephObjectStore’s internal certificate (the one that’s specified under spec.gateway.sslCertificateRef).