Kubernetes Cluster API

Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.

Started by the Kubernetes Special Interest Group (SIG) Cluster Lifecycle, the Cluster API project uses Kubernetes-style APIs and patterns to automate cluster lifecycle management for platform operators. The supporting infrastructure, like virtual machines, networks, load balancers, and VPCs, as well as the Kubernetes cluster configuration are all defined in the same way that application developers operate deploying and managing their workloads. This enables consistent and repeatable cluster deployments across a wide variety of infrastructure environments.

Getting started

• Quick Start
• Concepts
• Developer guide
• Contributing
• Videos explaining Cluster API architecture

Why build Cluster API?

Kubernetes is a complex system that relies on several components being configured correctly to have a working cluster. Recognizing this as a potential stumbling block for users, the community focused on simplifying the bootstrapping process. Today, over 100 Kubernetes distributions and installers have been created, each with different default configurations for clusters and supported infrastructure providers. SIG Cluster Lifecycle saw a need for a single tool to address a set of common overlapping installation concerns and started kubeadm.

Kubeadm was designed as a focused tool for bootstrapping a best-practices Kubernetes cluster. The core tenet behind the kubeadm project was to create a tool that other installers can leverage and ultimately alleviate the amount of configuration that an individual installer needed to maintain. Since it began, kubeadm has become the underlying bootstrapping tool for several other applications, including Kubespray, minikube, kind, etc.

However, while kubeadm and other bootstrap providers reduce installation complexity, they don’t address how to manage a cluster day-to-day or a Kubernetes environment long term. You are still faced with several questions when setting up a production environment, including:

• How can I consistently provision machines, load balancers, VPC, etc., across multiple infrastructure providers and locations?
• How can I automate cluster lifecycle management, including things like upgrades and cluster deletion?
• How can I scale these processes to manage any number of clusters?

SIG Cluster Lifecycle began the Cluster API project as a way to address these gaps by building declarative, Kubernetes-style APIs, that automate cluster creation, configuration, and management. Using this model, Cluster API can also be extended to support any infrastructure provider (AWS, Azure, vSphere, etc.) or bootstrap provider (kubeadm is default) you need. See the growing list of available providers.

Goals

• To manage the lifecycle (create, scale, upgrade, destroy) of Kubernetes-conformant clusters using a declarative API.
• To work in different environments, both on-premises and in the cloud.
• To define common operations, provide a default implementation, and provide the ability to swap out implementations for alternative ones.
• To reuse and integrate existing ecosystem components rather than duplicating their functionality (e.g. node-problem-detector, cluster autoscaler, SIG-Multi-cluster).
• To provide a transition path for Kubernetes lifecycle products to adopt Cluster API incrementally. Specifically, existing cluster lifecycle management tools should be able to adopt Cluster API in a staged manner, over the course of multiple releases, or even adopting a subset of Cluster API.

Non-goals

• To add these APIs to Kubernetes core (kubernetes/kubernetes).
  • This API should live in a namespace outside the core and follow the best practices defined by api-reviewers, but is not subject to core-api constraints.
• To manage the lifecycle of infrastructure unrelated to the running of Kubernetes-conformant clusters.
• To force all Kubernetes lifecycle products (kOps, Kubespray, GKE, AKS, EKS, IKS etc.) to support or use these APIs.
• To manage non-Cluster API provisioned Kubernetes-conformant clusters.
• To manage a single cluster spanning multiple infrastructure providers.
• To configure a machine at any time other than create or upgrade.
• To duplicate functionality that exists or is coming to other tooling, e.g., updating kubelet configuration (c.f. dynamic kubelet configuration), or updating apiserver, controller-manager, scheduler configuration (c.f. component-config effort) after the cluster is deployed.

🤗 Community, discussion, contribution, and support

Cluster API is developed in the open, and is constantly being improved by our users, contributors, and maintainers. It is because of you that we are able to automate cluster lifecycle management for the community. Join us!

If you have questions or want to get the latest project news, you can connect with us in the following ways:

• Chat with us on the Kubernetes Slack in the #cluster-api channel
• Subscribe to the SIG Cluster Lifecycle Google Group for access to documents and calendars
• Join our Cluster API working group sessions where we share the latest project news, demos, answer questions, and triage issues
  • Weekly on Wednesdays @ 10:00 PT on Zoom
  • Previous meetings: [ notes | recordings ]

Pull Requests and feedback on issues are very welcome! See the issue tracker if you’re unsure where to start, especially the Good first issue and Help wanted tags, and also feel free to reach out to discuss.

See also our contributor guide and the Kubernetes community page for more details on how to get involved.

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

Quick Start

In this tutorial we’ll cover the basics of how to use Cluster API to create one or more Kubernetes clusters.

Installation

There are two major quickstart paths: Using clusterctl or the Cluster API Operator.

This article describes a path that uses the clusterctl CLI tool to handle the lifecycle of a Cluster API management cluster.

The clusterctl command line interface is specifically designed for providing a simple “day 1 experience” and a quick start with Cluster API. It automates fetching the YAML files defining provider components and installing them.

Additionally it encodes a set of best practices in managing providers, that helps the user in avoiding mis-configurations or in managing day 2 operations such as upgrades.

The Cluster API Operator is a Kubernetes Operator built on top of clusterctl and designed to empower cluster administrators to handle the lifecycle of Cluster API providers within a management cluster using a declarative approach. It aims to improve user experience in deploying and managing Cluster API, making it easier to handle day-to-day tasks and automate workflows with GitOps. Visit the CAPI Operator quickstart if you want to experiment with this tool.

Common Prerequisites

• Install and setup kubectl in your local environment
• Install kind and Docker
• Install Helm

Install and/or configure a Kubernetes cluster

Cluster API requires an existing Kubernetes cluster accessible via kubectl. During the installation process the Kubernetes cluster will be transformed into a management cluster by installing the Cluster API provider components, so it is recommended to keep it separated from any application workload.

It is a common practice to create a temporary, local bootstrap cluster which is then used to provision a target management cluster on the selected infrastructure provider.

Choose one of the options below:

1. Existing Management Cluster

   For production use-cases a “real” Kubernetes cluster should be used with appropriate backup and disaster recovery policies and procedures in place. The Kubernetes cluster must be at least v1.20.0.

   export KUBECONFIG=<...>
   

OR

2. Kind

   kind can be used for creating a local Kubernetes cluster for development environments or for the creation of a temporary bootstrap cluster used to provision a target management cluster on the selected infrastructure provider.

   The installation procedure depends on the version of kind; if you are planning to use the Docker infrastructure provider, please follow the additional instructions in the dedicated tab:

   DefaultDockerKubeVirt

   Create the kind cluster:

   kind create cluster
   

   Test to ensure the local kind cluster is ready:

   kubectl cluster-info
   

   Run the following command to create a kind config file for allowing the Docker provider to access Docker on the host:

   cat > kind-cluster-with-extramounts.yaml <<EOF
   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   networking:
     ipFamily: dual
   nodes:
   - role: control-plane
     extraMounts:
       - hostPath: /var/run/docker.sock
         containerPath: /var/run/docker.sock
   EOF
   

   Then follow the instruction for your kind version using kind create cluster --config kind-cluster-with-extramounts.yaml to create the management cluster using the above file.

   Create the Kind Cluster

   KubeVirt is a cloud native virtualization solution. The virtual machines we’re going to create and use for the workload cluster’s nodes, are actually running within pods in the management cluster. In order to communicate with the workload cluster’s API server, we’ll need to expose it. We are using Kind which is a limited environment. The easiest way to expose the workload cluster’s API server (a pod within a node running in a VM that is itself running within a pod in the management cluster, that is running inside a Docker container), is to use a LoadBalancer service.

   To allow using a LoadBalancer service, we can’t use the kind’s default CNI (kindnet), but we’ll need to install another CNI, like Calico. In order to do that, we’ll need first to initiate the kind cluster with two modifications:

   1. Disable the default CNI
   2. Add the Docker credentials to the cluster, to avoid the Docker Hub pull rate limit of the calico images; read more about it in the docker documentation, and in the kind documentation.

   Create a configuration file for kind. Please notice the Docker config file path, and adjust it to your local setting:

   cat <<EOF > kind-config.yaml
   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   networking:
   # the default CNI will not be installed
     disableDefaultCNI: true
   nodes:
   - role: control-plane
     extraMounts:
      - containerPath: /var/lib/kubelet/config.json
        hostPath: <YOUR DOCKER CONFIG FILE PATH>
   EOF
   

   Now, create the kind cluster with the configuration file:

   kind create cluster --config=kind-config.yaml
   

   Test to ensure the local kind cluster is ready:

   kubectl cluster-info
   

   Install the Calico CNI

   Now we’ll need to install a CNI. In this example, we’re using calico, but other CNIs should work as well. Please see calico installation guide for more details (use the “Manifest” tab). Below is an example of how to install calico version v3.29.1.

   Use the Calico manifest to create the required resources; e.g.:

   kubectl create -f  https://raw.githubusercontent.com/projectcalico/calico/v3.29.1/manifests/calico.yaml
   

Install clusterctl

The clusterctl CLI tool handles the lifecycle of a Cluster API management cluster.

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.14.0/clusterctl-linux-amd64 -o clusterctl

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.14.0/clusterctl-linux-arm64 -o clusterctl

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.14.0/clusterctl-linux-ppc64le -o clusterctl

sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl

clusterctl version

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.14.0/clusterctl-darwin-amd64 -o clusterctl

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.14.0/clusterctl-darwin-arm64 -o clusterctl

chmod +x ./clusterctl

sudo mv ./clusterctl /usr/local/bin/clusterctl

clusterctl version

brew install clusterctl

clusterctl version

curl.exe -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.14.0/clusterctl-windows-amd64.exe -o clusterctl.exe

clusterctl.exe version

Initialize the management cluster

Now that we’ve got clusterctl installed and all the prerequisites in place, let’s transform the Kubernetes cluster into a management cluster by using clusterctl init.

The command accepts as input a list of providers to install; when executed for the first time, clusterctl init automatically adds to the list the cluster-api core provider, and if unspecified, it also adds the kubeadm bootstrap and kubeadm control-plane providers.

Enabling Feature Gates

Feature gates can be enabled by exporting environment variables before executing clusterctl init. For example, the ClusterTopology feature, which is required to enable support for managed topologies and ClusterClass, can be enabled via:

export CLUSTER_TOPOLOGY=true

Additional documentation about experimental features can be found in Experimental Features.

Initialization for common providers

Depending on the infrastructure provider you are planning to use, some additional prerequisites should be satisfied before getting started with Cluster API. See below for the expected settings for common providers.

export LINODE_TOKEN=<your-access-token>

# Initialize the management cluster
clusterctl init --infrastructure linode-linode

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v2.13.0/clusterawsadm-linux-amd64 -o clusterawsadm

chmod +x clusterawsadm

sudo mv clusterawsadm /usr/local/bin

clusterawsadm version

export AWS_REGION=us-east-1 # This is used to help encode your environment variables
export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
export AWS_SESSION_TOKEN=<session-token> # If you are using Multi-Factor Auth.

# The clusterawsadm utility takes the credentials that you set as environment
# variables and uses them to create a CloudFormation stack in your AWS account
# with the correct IAM resources.
clusterawsadm bootstrap iam create-cloudformation-stack

# Create the base64 encoded credentials using clusterawsadm.
# This command uses your environment variables and encodes
# them in a value to be stored in a Kubernetes Secret.
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

# Finally, initialize the management cluster
clusterctl init --infrastructure aws

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v2.13.0/clusterawsadm-darwin-amd64 -o clusterawsadm

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v2.13.0/clusterawsadm-darwin-arm64 -o clusterawsadm

chmod +x clusterawsadm

sudo mv clusterawsadm /usr/local/bin

clusterawsadm version

export AWS_REGION=us-east-1 # This is used to help encode your environment variables
export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
export AWS_SESSION_TOKEN=<session-token> # If you are using Multi-Factor Auth.

# The clusterawsadm utility takes the credentials that you set as environment
# variables and uses them to create a CloudFormation stack in your AWS account
# with the correct IAM resources.
clusterawsadm bootstrap iam create-cloudformation-stack

# Create the base64 encoded credentials using clusterawsadm.
# This command uses your environment variables and encodes
# them in a value to be stored in a Kubernetes Secret.
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

# Finally, initialize the management cluster
clusterctl init --infrastructure aws

brew install clusterawsadm

clusterawsadm version

export AWS_REGION=us-east-1 # This is used to help encode your environment variables
export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
export AWS_SESSION_TOKEN=<session-token> # If you are using Multi-Factor Auth.

# The clusterawsadm utility takes the credentials that you set as environment
# variables and uses them to create a CloudFormation stack in your AWS account
# with the correct IAM resources.
clusterawsadm bootstrap iam create-cloudformation-stack

# Create the base64 encoded credentials using clusterawsadm.
# This command uses your environment variables and encodes
# them in a value to be stored in a Kubernetes Secret.
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

# Finally, initialize the management cluster
clusterctl init --infrastructure aws

curl.exe -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v2.13.0/clusterawsadm-windows-amd64.exe -o clusterawsadm.exe

clusterawsadm.exe version

$Env:AWS_REGION="us-east-1" # This is used to help encode your environment variables
$Env:AWS_ACCESS_KEY_ID="<your-access-key>"
$Env:AWS_SECRET_ACCESS_KEY="<your-secret-access-key>"
$Env:AWS_SESSION_TOKEN="<session-token>" # If you are using Multi-Factor Auth.

# The clusterawsadm utility takes the credentials that you set as environment
# variables and uses them to create a CloudFormation stack in your AWS account
# with the correct IAM resources.
clusterawsadm bootstrap iam create-cloudformation-stack

# Create the base64 encoded credentials using clusterawsadm.
# This command uses your environment variables and encodes
# them in a value to be stored in a Kubernetes Secret.
$Env:AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

# Finally, initialize the management cluster
clusterctl init --infrastructure aws

export AZURE_SUBSCRIPTION_ID="<SubscriptionId>"

# Create an Azure Service Principal and paste the output here
export AZURE_TENANT_ID="<Tenant>"
export AZURE_CLIENT_ID="<AppId>"
export AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY=$AZURE_CLIENT_ID # for compatibility with CAPZ v1.16 templates
export AZURE_CLIENT_SECRET="<Password>"

# Settings needed for AzureClusterIdentity used by the AzureCluster
export AZURE_CLUSTER_IDENTITY_SECRET_NAME="cluster-identity-secret"
export CLUSTER_IDENTITY_NAME="cluster-identity"
export AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE="default"

# Create a secret to include the password of the Service Principal identity created in Azure
# This secret will be referenced by the AzureClusterIdentity used by the AzureCluster
kubectl create secret generic "${AZURE_CLUSTER_IDENTITY_SECRET_NAME}" --from-literal=clientSecret="${AZURE_CLIENT_SECRET}" --namespace "${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}"

# Finally, initialize the management cluster
clusterctl init --infrastructure azure

# The cloudscale API token.
# You may want to set this in `$XDG_CONFIG_HOME/cluster-api/clusterctl.yaml` so your token is not in
# bash history
export CLOUDSCALE_API_TOKEN="AAAEXAMPLE"

# initialize the management cluster
clusterctl init --infrastructure cloudscale-ch-cloudscale

[Global]
api-url = <cloudstackApiUrl>
api-key = <cloudstackApiKey>
secret-key = <cloudstackSecretKey>

export CLOUDSTACK_B64ENCODED_SECRET=`cat cloud-config | base64 | tr -d '\n'`

clusterctl init --infrastructure cloudstack

export DIGITALOCEAN_ACCESS_TOKEN=<your-access-token>
export DO_B64ENCODED_CREDENTIALS="$(echo -n "${DIGITALOCEAN_ACCESS_TOKEN}" | base64 | tr -d '\n')"

# Initialize the management cluster
clusterctl init --infrastructure digitalocean

# Enable the experimental Cluster topology feature.
export CLUSTER_TOPOLOGY=true

# Initialize the management cluster
clusterctl init --infrastructure docker

# Create the base64 encoded credentials by catting your credentials json.
# This command uses your environment variables and encodes
# them in a value to be stored in a Kubernetes Secret.
export GCP_B64ENCODED_CREDENTIALS=$( cat /path/to/gcp-credentials.json | base64 | tr -d '\n' )

# Finally, initialize the management cluster
clusterctl init --infrastructure gcp

clusterctl init --infrastructure harvester-harvester

# Please ensure that the values for `CLOUD_SDK_AK` and `CLOUD_SDK_SK` are base64 encoded.
export CLOUD_SDK_AK=$( echo $AccessKey | base64 | tr -d '\n' )
export CLOUD_SDK_SK=$( echo $SecretKey | base64 | tr -d '\n' )

# Finally, initialize the management cluster
clusterctl init --infrastructure huawei

export IBMCLOUD_API_KEY=<you_api_key>

# Finally, initialize the management cluster
clusterctl init --infrastructure ibmcloud

clusterctl init --infrastructure ionoscloud-ionoscloud

# Initialize the management cluster
clusterctl init --infrastructure k0sproject-k0smotron

# Initialize the management cluster
clusterctl init --bootstrap kairos-io --control-plane kairos-io --infrastructure kairos-io-fleet

# Initialize the management cluster
clusterctl init --infrastructure kubekey

# Initialize the management cluster
clusterctl init --infrastructure kubeswift-io

METALLB_VER=$(curl "https://api.github.com/repos/metallb/metallb/releases/latest" | jq -r ".tag_name")
kubectl apply -f "https://raw.githubusercontent.com/metallb/metallb/${METALLB_VER}/config/manifests/metallb-native.yaml"
kubectl wait pods -n metallb-system -l app=metallb,component=controller --for=condition=Ready --timeout=10m
kubectl wait pods -n metallb-system -l app=metallb,component=speaker --for=condition=Ready --timeout=2m

SUBNET=$(docker network inspect kind | jq -r 'first(.[0].IPAM.Config[].Subnet | select(test("^[0-9]+\\.")))')
PREFIX=$(echo $SUBNET | sed -E 's|^([0-9]+\.[0-9]+)\..*$|\1|g')

cat <<EOF | kubectl apply -f -
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: capi-ip-pool
  namespace: metallb-system
spec:
  addresses:
  - ${PREFIX}.255.200-${PREFIX}.255.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: empty
  namespace: metallb-system
EOF

# get KubeVirt version
KV_VER=$(curl "https://api.github.com/repos/kubevirt/kubevirt/releases/latest" | jq -r ".tag_name")
# deploy required CRDs
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${KV_VER}/kubevirt-operator.yaml"
# deploy the KubeVirt custom resource
kubectl apply -f "https://github.com/kubevirt/kubevirt/releases/download/${KV_VER}/kubevirt-cr.yaml"
kubectl wait -n kubevirt kv kubevirt --for=condition=Available --timeout=10m

clusterctl init --infrastructure kubevirt

clusterctl init --infrastructure metal-stack

# Initialize the management cluster
clusterctl init --infrastructure opennebula

# Install ORC (needed for CAPO >=v0.12)
kubectl apply -f https://github.com/k-orc/openstack-resource-controller/releases/latest/download/install.yaml
# Initialize the management cluster
clusterctl init --infrastructure openstack

export OSC_SECRET_KEY=<your-secret-key>
export OSC_ACCESS_KEY=<your-access-key>
export OSC_REGION=<you-region>
# Create namespace
kubectl create namespace cluster-api-provider-outscale-system
# Create secret
kubectl create secret generic cluster-api-provider-outscale --from-literal=access_key=${OSC_ACCESS_KEY} --from-literal=secret_key=${OSC_SECRET_KEY} --from-literal=region=${OSC_REGION}  -n cluster-api-provider-outscale-system
# Initialize the management cluster
clusterctl init --infrastructure outscale

export OXIDE_HOST=<silo host>
export OXIDE_TOKEN=<token>
# Create secret
kubectl create secret generic cluster-api-provider-oxide \
  --from-literal=oxide-host=${OXIDE_HOST} \
  --from-literal=oxide-token=${OXIDE_TOKEN}
# Initialize the management cluster
clusterctl init --infrastructure oxide

# The host for the Proxmox cluster
export PROXMOX_URL="https://pve.example:8006"
# The Proxmox token ID to access the remote Proxmox endpoint
export PROXMOX_TOKEN='root@pam!capi'
# The secret associated with the token ID
# You may want to set this in `$XDG_CONFIG_HOME/cluster-api/clusterctl.yaml` so your password is not in
# bash history
export PROXMOX_SECRET="1234-1234-1234-1234"


# Finally, initialize the management cluster
clusterctl init --infrastructure proxmox --ipam in-cluster

# Initialize the management cluster
clusterctl init --infrastructure scaleway

# Initialize the management cluster
clusterctl init --infrastructure vcd

clusterctl init --infrastructure vcluster

# Initialize the management cluster
clusterctl init --infrastructure virtink

# The username used to access the remote vSphere endpoint
export VSPHERE_USERNAME="vi-admin@vsphere.local"
# The password used to access the remote vSphere endpoint
# You may want to set this in `$XDG_CONFIG_HOME/cluster-api/clusterctl.yaml` so your password is not in
# bash history
export VSPHERE_PASSWORD="admin!23"

# Finally, initialize the management cluster
clusterctl init --infrastructure vsphere

export VULTR_API_KEY="$(echo -n "${VULTR_API_KEY}" | base64 | tr -d '\n')"

# initialize the management cluster
clusterctl init --infrastructure vultr-vultr

The output of clusterctl init is similar to this:

Fetching providers
Installing cert-manager Version="v1.11.0"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v1.0.0" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v1.0.0" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v1.0.0" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-docker" Version="v1.0.0" TargetNamespace="capd-system"

Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -

Create your first workload cluster

Once the management cluster is ready, you can create your first workload cluster.

Preparing the workload cluster configuration

The clusterctl generate cluster command returns a YAML template for creating a workload cluster.

Required configuration for common providers

Depending on the infrastructure provider you are planning to use, some additional prerequisites should be satisfied before configuring a cluster with Cluster API. Instructions are provided for common providers below.

Otherwise, you can look at the clusterctl generate cluster command documentation for details about how to discover the list of variables required by a cluster templates.

export LINODE_REGION=us-ord
export LINODE_TOKEN=<your linode PAT>
export LINODE_CONTROL_PLANE_MACHINE_TYPE=g6-standard-2
export LINODE_MACHINE_TYPE=g6-standard-2

export AWS_REGION=us-east-1
export AWS_SSH_KEY_NAME=default
# Select instance types
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3.large
export AWS_NODE_MACHINE_TYPE=t3.large

# Name of the Azure datacenter location. Change this value to your desired location.
export AZURE_LOCATION="centralus"

# Select VM types.
export AZURE_CONTROL_PLANE_MACHINE_TYPE="Standard_D2s_v3"
export AZURE_NODE_MACHINE_TYPE="Standard_D2s_v3"

# [Optional] Select resource group. The default value is ${CLUSTER_NAME}.
export AZURE_RESOURCE_GROUP="<ResourceGroupName>"

# The cloudscale API token.
# You may want to set this in `$XDG_CONFIG_HOME/cluster-api/clusterctl.yaml` so your token is not in
# bash history
export CLOUDSCALE_API_TOKEN="AAAEXAMPLE"
# SSH public key added to nodes
export CLOUDSCALE_SSH_PUBLIC_KEY="ssh-ed25519 AAAA..."
# cloudscale.ch region
export CLOUDSCALE_REGION="lpg"
# Server image for nodes
export CLOUDSCALE_MACHINE_IMAGE="custom:ubuntu-2404-kube-v1.36.1"
# Flavor for control plane nodes
export CLOUDSCALE_CONTROL_PLANE_MACHINE_FLAVOR="flex-4-2"
# Flavor for worker nodes 
export CLOUDSCALE_WORKER_MACHINE_FLAVOR="flex-4-2"
# Root volume size in GB
export CLOUDSCALE_ROOT_VOLUME_SIZE="50"

clusterctl generate cluster --infrastructure cloudstack --list-variables capi-quickstart

# Set this to the name of the zone in which to deploy the cluster
export CLOUDSTACK_ZONE_NAME=<zone name>
# The name of the network on which the VMs will reside
export CLOUDSTACK_NETWORK_NAME=<network name>
# The endpoint of the workload cluster
export CLUSTER_ENDPOINT_IP=<cluster endpoint address>
export CLUSTER_ENDPOINT_PORT=<cluster endpoint port>
# The service offering of the control plane nodes
export CLOUDSTACK_CONTROL_PLANE_MACHINE_OFFERING=<control plane service offering name>
# The service offering of the worker nodes
export CLOUDSTACK_WORKER_MACHINE_OFFERING=<worker node service offering name>
# The capi compatible template to use
export CLOUDSTACK_TEMPLATE_NAME=<template name>
# The ssh key to use to log into the nodes
export CLOUDSTACK_SSH_KEY_NAME=<ssh key name>

export DO_REGION=nyc1
export DO_SSH_KEY_FINGERPRINT=<your-ssh-key-fingerprint>
export DO_CONTROL_PLANE_MACHINE_TYPE=s-2vcpu-2gb
export DO_CONTROL_PLANE_MACHINE_IMAGE=<your-capi-image-id>
export DO_NODE_MACHINE_TYPE=s-2vcpu-2gb
export DO_NODE_MACHINE_IMAGE==<your-capi-image-id>

# The list of service CIDR, default ["10.128.0.0/12"]
export SERVICE_CIDR=["10.96.0.0/12"]

# The list of pod CIDR, default ["192.168.0.0/16"]
export POD_CIDR=["192.168.0.0/16"]

# The service domain, default "cluster.local"
export SERVICE_DOMAIN="k8s.test"

export POD_SECURITY_STANDARD_ENABLED="false"

# Name of the GCP datacenter location. Change this value to your desired location
export GCP_REGION="<GCP_REGION>"
export GCP_PROJECT="<GCP_PROJECT>"
# Make sure to use same Kubernetes version here as building the GCE image
export KUBERNETES_VERSION=1.23.3
# This is the image you built. See https://github.com/kubernetes-sigs/image-builder
export IMAGE_ID=projects/$GCP_PROJECT/global/images/<built image>
export GCP_CONTROL_PLANE_MACHINE_TYPE=n1-standard-2
export GCP_NODE_MACHINE_TYPE=n1-standard-2
export GCP_NETWORK_NAME=<GCP_NETWORK_NAME or default>
export CLUSTER_NAME="<CLUSTER_NAME>"

# Cloud Provider credentials, which are a Kubeconfig generated using this process: https://docs.harvesterhci.io/v1.3/rancher/cloud-provider/#deploying-to-the-rke2-custom-cluster-experimental
# Since v0.1.5, this can be left "", because the controller can update it automatically
export CLOUD_CONFIG_KUBECONFIG_B64=""
# Name of the CAPI Cluster
export CLUSTER_NAME="<CLUSTER_NAME>"
# Number of Control Plane machines
export CONTROL_PLANE_MACHINE_COUNT=3
# URL to access the Harvester Cluster, this will be overridden by the controller
export HARVESTER_ENDPOINT=""
# Base64-Encoded Kubeconfig to access Harvester, which can be downloaded from Harvester's UI or from a Harvester Manager Node.
export HARVESTER_KUBECONFIG_B64="<HARVESTER_KUBECONFIG_ENCODED_IN_BASE64>"
# Namespace for all resources in the Management Cluster
export NAMESPACE="test"
# Pod CIDR for the Workload Cluster, it should have the format: 192.168.0.0/16
export POD_CIDR="10.42.0.0/16"
# Service CIDR for the Workload Cluster, it should have the format : 192.168.0.0/16 and be different from POD_CIDR
export SERVICE_CIDR="10.43.0.0/16"
# Reference to SSH Keypair in Harvester. It should follow the format <NAMESPACE>/<NAME>
export SSH_KEYPAIR="default/ssk-key-pair"
# Namespace in Harvester where the VMs will be created.
export TARGET_HARVESTER_NAMESPACE="default"
# Disk Size to be used by the VMs
export VM_DISK_SIZE="50Gi"
# Reference to OS Image in Harvester which will be used for creating VMs, It must follow the format <NAMESPACE>/<NAME>
export VM_IMAGE_NAME="default/jammy-server"
# Reference to VM Network in Harvester. It must follow the format <NAMESPACE>/<NAME>
export VM_NETWORK="default/untagged"
# Linux Username for the VMs
export VM_SSH_USER="ubuntu"
# Number of Worker nodes in the target Workload cluster
export WORKER_MACHINE_COUNT=2

# huawei cloud region
export HC_REGION="cn-east-1"
# ECS SSH key name
export HC_SSH_KEY_NAME="default"
# kubernetes version
export KUBERNETES_VERSION="1.32.0"
# number of control plane machines
export CONTROL_PLANE_MACHINE_COUNT="1"
# number of worker machines
export WORKER_MACHINE_COUNT="1"
# control plane machine type
export HC_CONTROL_PLANE_MACHINE_TYPE="x1e.2u.4g"
# worker node machine type
export HC_NODE_MACHINE_TYPE="x1e.2u.4g"
# ECS image ID
export ECS_IMAGE_ID="218ca5t7-bxf3-5dg0-852p-y703c9fe1a52"

# Required environment variables for VPC
# VPC region
export IBMVPC_REGION=us-south
# VPC zone within the region
export IBMVPC_ZONE=us-south-1
# ID of the resource group in which the VPC will be created
export IBMVPC_RESOURCEGROUP=<your-resource-group-id>
# Name of the VPC
export IBMVPC_NAME=ibm-vpc-0
export IBMVPC_IMAGE_ID=<you-image-id>
# Profile for the virtual server instances
export IBMVPC_PROFILE=bx2-4x16
export IBMVPC_SSHKEY_ID=<your-sshkey-id>

# Required environment variables for PowerVS
export IBMPOWERVS_SSHKEY_NAME=<your-ssh-key>
# Internal and external IP of the network
export IBMPOWERVS_VIP=<internal-ip>
export IBMPOWERVS_VIP_EXTERNAL=<external-ip>
export IBMPOWERVS_VIP_CIDR=29
export IBMPOWERVS_IMAGE_NAME=<your-capi-image-name>
# ID of the PowerVS service instance
export IBMPOWERVS_SERVICE_INSTANCE_ID=<service-instance-id>
export IBMPOWERVS_NETWORK_NAME=<your-capi-network-name>

# The token which is used to authenticate against the IONOS Cloud API
export IONOS_TOKEN=<your-token>
# The datacenter ID where the cluster will be deployed
export IONOSCLOUD_DATACENTER_ID="<your-datacenter-id>"
# The IP of the control plane endpoint
export CONTROL_PLANE_ENDPOINT_IP=10.10.10.4
# The location of the data center where the cluster will be deployed
export CONTROL_PLANE_ENDPOINT_LOCATION=de/txl
# The image ID of the custom image that will be used for the VMs
export IONOSCLOUD_MACHINE_IMAGE_ID="<your-image-id>"
# The SSH key that will be used to access the VMs
export IONOSCLOUD_MACHINE_SSH_KEYS="<your-ssh-key>"

export CONTROL_PLANE_ENDPOINT_HOST=<your-control-plane-host>
export AURORABOOT_URL=http://<auroraboot-host>:8080

# Required environment variables
# The KKZONE is used to specify where to download the binaries. (e.g. "", "cn")
export KKZONE=""
# The ssh name of the all instance Linux user. (e.g. root, ubuntu)
export USER_NAME=<your-linux-user>
# The ssh password of the all instance Linux user.
export PASSWORD=<your-linux-user-password>
# The ssh IP address of the all instance. (e.g. "[{address: 192.168.100.3}, {address: 192.168.100.4}]")
export INSTANCES=<your-linux-ip-address>
# The cluster control plane VIP. (e.g. "192.168.100.100")
export CONTROL_PLANE_ENDPOINT_IP=<your-control-plane-virtual-ip>

export KUBESWIFT_IMAGE="ubuntu-noble"
export KUBESWIFT_CONTROL_PLANE_CLASS="capi-controlplane"
export KUBESWIFT_WORKER_CLASS="capi-worker"

export NODE_VM_IMAGE_TEMPLATE="quay.io/capk/ubuntu-2404-container-disk:v1.32.1"
export CAPK_GUEST_K8S_VERSION="${NODE_VM_IMAGE_TEMPLATE/*:/}"
export CRI_PATH="unix:///var/run/containerd/containerd.sock"

# The URL of the kernel to deploy.
export DEPLOY_KERNEL_URL="http://172.22.0.1:6180/images/ironic-python-agent.kernel"
# The URL of the ramdisk to deploy.
export DEPLOY_RAMDISK_URL="http://172.22.0.1:6180/images/ironic-python-agent.initramfs"
# The URL of the Ironic endpoint.
export IRONIC_URL="http://172.22.0.1:6385/v1/"
# The URL of the Ironic inspector endpoint.
export IRONIC_INSPECTOR_URL="http://172.22.0.1:5050/v1/"
# Do not use a dedicated CA certificate for Ironic API. Any value provided in this variable disables additional CA certificate validation.
# To provide a CA certificate, leave this variable unset. If unset, then IRONIC_CA_CERT_B64 must be set.
export IRONIC_NO_CA_CERT=true
# Disables basic authentication for Ironic API. Any value provided in this variable disables authentication.
# To enable authentication, leave this variable unset. If unset, then IRONIC_USERNAME and IRONIC_PASSWORD must be set.
export IRONIC_NO_BASIC_AUTH=true
# Disables basic authentication for Ironic inspector API. Any value provided in this variable disables authentication.
# To enable authentication, leave this variable unset. If unset, then IRONIC_INSPECTOR_USERNAME and IRONIC_INSPECTOR_PASSWORD must be set.
export IRONIC_INSPECTOR_NO_BASIC_AUTH=true

export METAL_PARTITION=<metal-stack-partition>
export METAL_PROJECT_ID=<metal-stack-project-id>
export CONTROL_PLANE_IP=<metal-stack-control-plane-ip>

export FIREWALL_MACHINE_IMAGE=<firewall-os-image>
export FIREWALL_MACHINE_SIZE=<firewall-size>

export CONTROL_PLANE_MACHINE_IMAGE=<machine-os-image>
export CONTROL_PLANE_MACHINE_SIZE=<machine-size>
export WORKER_MACHINE_IMAGE=<machine-os-image>
export WORKER_MACHINE_SIZE=<machine-size>

clusterctl generate cluster --infrastructure nutanix --list-variables capi-quickstart

# OpenNebula API endpoint and credentials
export ONE_XMLRPC='http://10.2.11.40:2633/RPC2'
export ONE_AUTH='oneadmin:opennebula'

# VM and VR templates to construct workload clusters from
export MACHINE_TEMPLATE_NAME='capone131'
export ROUTER_TEMPLATE_NAME='capone131-vr'

# VNs to deploy workload clusters into
export PUBLIC_NETWORK_NAME='service'
export PRIVATE_NETWORK_NAME='private'

# Name of the new workload cluster
export CLUSTER_NAME='one'

# Cloud-Provider image to deploy inside the new workload cluster
export CCM_IMG='ghcr.io/opennebula/cloud-provider-opennebula:latest'

# Initial size of the new workload cluster
export CONTROL_PLANE_MACHINE_COUNT='1'
export WORKER_MACHINE_COUNT='1'

clusterctl generate cluster --infrastructure openstack --list-variables capi-quickstart

wget https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-openstack/master/templates/env.rc -O /tmp/env.rc
source /tmp/env.rc <path/to/clouds.yaml> <cloud>

# The list of nameservers for OpenStack Subnet being created.
# Set this value when you need create a new network/subnet while the access through DNS is required.
export OPENSTACK_DNS_NAMESERVERS=<dns nameserver>
# FailureDomain is the failure domain the machine will be created in.
export OPENSTACK_FAILURE_DOMAIN=<availability zone name>
# The flavor reference for the flavor for your server instance.
export OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR=<flavor>
# The flavor reference for the flavor for your server instance.
export OPENSTACK_NODE_MACHINE_FLAVOR=<flavor>
# The name of the image to use for your server instance. If the RootVolume is specified, this will be ignored and use rootVolume directly.
export OPENSTACK_IMAGE_NAME=<image name>
# The SSH key pair name
export OPENSTACK_SSH_KEY_NAME=<ssh key pair name>
# The external network
export OPENSTACK_EXTERNAL_NETWORK_ID=<external network ID>

# The outscale root disk iops
export OSC_IOPS="<IOPS>"
# The outscale root disk size
export OSC_VOLUME_SIZE="<VOLUME_SIZE>"
# The outscale root disk volumeType
export OSC_VOLUME_TYPE="<VOLUME_TYPE>"
# The outscale key pair
export OSC_KEYPAIR_NAME="<KEYPAIR_NAME>"
# The outscale subregion name
export OSC_SUBREGION_NAME="<SUBREGION_NAME>"
# The outscale vm type
export OSC_VM_TYPE="<VM_TYPE>"
# The outscale image name
export OSC_IMAGE_NAME="<IMAGE_NAME>"

## 1. Upload an Ubuntu 24.04 base image to your Oxide Silo
## 2. Export the base image ID (Ubuntu 24.04)
export OXIDE_BOOT_DISK_IMAGE_ID="<ubuntu-24-04-id>"
## 3. Set the Project for packer to build in
export OXIDE_PROJECT="<project>"
## 4. clone image-builder
git clone https://github.com/kubernetes-sigs/image-builder.git
## 5. Build CAPI image for Oxide
cd image-builder/images/capi && make build-oxide-ubuntu-2404

# The Oxide Project to deploy to (required)
export OXIDE_PROJECT="<Project>"
# The Oxide Image built from image-builder (required)
export OXIDE_IMAGE_ID="<image-id from image-builder result>"
# The Oxide VPC to use
export OXIDE_VPC="default"

## To discover additional optional environment variables and their defaults run: 
clusterctl generate cluster capi-quickstart --list-variables

OXIDE_RULES_FILE="$(mktemp)"

oxide vpc firewall-rules view --project "$OXIDE_PROJECT" --vpc "$OXIDE_VPC" \
| jq --arg vpc "$OXIDE_VPC" --arg name "allow-kube-apiserver" '{
    rules: (
      [ .rules[]
        | select(.name != $name)
        | {action, description, direction, filters, name, priority, status, targets} ]
      + [ {
          name: $name,
          description: "Allow kube-apiserver (TCP 6443) from anywhere (https://github.com/oxidecomputer/cluster-api-provider-oxide/docs/getting-started.md)",
          action: "allow",
          direction: "inbound",
          priority: 0,
          status: "enabled",
          filters: { protocols: [ {type:"tcp"} ], ports: ["6443"] },
          targets: [ { type: "vpc", value: $vpc } ]
        } ]
    )
  }' > "$OXIDE_RULES_FILE"


oxide vpc firewall-rules update --project "$OXIDE_PROJECT" --vpc "$OXIDE_VPC" --json-body "$OXIDE_RULES_FILE"

# The node that hosts the VM template to be used to provision VMs
export PROXMOX_SOURCENODE="pve"
# The template VM ID used for cloning VMs
export TEMPLATE_VMID=100
# The ssh authorized keys used to ssh to the machines.
export VM_SSH_KEYS="ssh-ed25519 ..., ssh-ed25519 ..."
# The IP address used for the control plane endpoint
export CONTROL_PLANE_ENDPOINT_IP=10.10.10.4
# The IP ranges for Cluster nodes
export NODE_IP_RANGES="[10.10.10.5-10.10.10.50, 10.10.10.55-10.10.10.70]"
# The gateway for the machines network-config.
export GATEWAY="10.10.10.1"
# Subnet Mask in CIDR notation for your node IP ranges
export IP_PREFIX=24
# The Proxmox network device for VMs
export BRIDGE="vmbr1"
# The dns nameservers for the machines network-config.
export DNS_SERVERS="[8.8.8.8,8.8.4.4]"
# The Proxmox nodes used for VM deployments
export ALLOWED_NODES="[pve1,pve2,pve3]"

# Scaleway credentials, project ID and region.
export SCW_ACCESS_KEY="<ACCESS_KEY>"
export SCW_SECRET_KEY="<SECRET_KEY>"
export SCW_PROJECT_ID="<PROJECT_ID>"
export SCW_REGION="fr-par"

# Scaleway Instance image names that will be used to provision servers.
export CONTROL_PLANE_MACHINE_IMAGE="<IMAGE_NAME>"
export WORKER_MACHINE_IMAGE="<IMAGE_NAME>"

export TINKERBELL_IP=<hegel ip>

clusterctl generate cluster --infrastructure vcd --list-variables capi-quickstart

export CLUSTER_NAME=kind
export CLUSTER_NAMESPACE=vcluster
export VCLUSTER_YAML=""
export KUBERNETES_VERSION=1.23.4
export HELM_VALUES="service:\n  type: NodePort"

clusterctl generate cluster --infrastructure virtink --list-variables capi-quickstart

# The vCenter server IP or FQDN
export VSPHERE_SERVER="10.0.0.1"
# The vSphere datacenter to deploy the management cluster on
export VSPHERE_DATACENTER="SDDC-Datacenter"
# The vSphere datastore to deploy the management cluster on
export VSPHERE_DATASTORE="vsanDatastore"
# The VM network to deploy the management cluster on
export VSPHERE_NETWORK="VM Network"
# The vSphere resource pool for your VMs
export VSPHERE_RESOURCE_POOL="*/Resources"
# The VM folder for your VMs. Set to "" to use the root vSphere folder
export VSPHERE_FOLDER="vm"
# The VM template to use for your VMs
export VSPHERE_TEMPLATE="ubuntu-1804-kube-v1.17.3"
# The public ssh authorized key on all machines
export VSPHERE_SSH_AUTHORIZED_KEY="ssh-rsa AAAAB3N..."
# The certificate thumbprint for the vCenter server
export VSPHERE_TLS_THUMBPRINT="97:48:03:8D:78:A9..."
# The storage policy to be used (optional). Set to "" if not required
export VSPHERE_STORAGE_POLICY="policy-one"
# The IP address used for the control plane endpoint
export CONTROL_PLANE_ENDPOINT_IP="1.2.3.4"

export CLUSTER_NAME=<clustername>
export KUBERNETES_VERSION=v1.28.9
export CONTROL_PLANE_MACHINE_COUNT=1
export CONTROL_PLANE_PLANID=<plan_id>
export WORKER_MACHINE_COUNT=1
export WORKER_PLANID=<plan_id>
export MACHINE_IMAGE=<snapshot_id>
export REGION=<region>
export PLANID=<plan_id>
export VPCID=<vpc_id>
export SSHKEY_ID=<sshKey_id>

Generating the cluster configuration

For the purpose of this tutorial, we’ll name our cluster capi-quickstart.

clusterctl generate cluster capi-quickstart --flavor development \
  --kubernetes-version v1.36.1 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

export CLUSTER_NAME=kind
export CLUSTER_NAMESPACE=vcluster
export VCLUSTER_YAML=""
export KUBERNETES_VERSION=1.31.2
export HELM_VALUES="service:\n  type: NodePort"

kubectl create namespace ${CLUSTER_NAMESPACE}
clusterctl generate cluster ${CLUSTER_NAME} \
    --infrastructure vcluster \
    --kubernetes-version ${KUBERNETES_VERSION} \
    --target-namespace ${CLUSTER_NAMESPACE} > capi-quickstart.yaml

clusterctl generate cluster capi-quickstart \
  --infrastructure="kubevirt" \
  --flavor lb \
  --kubernetes-version ${CAPK_GUEST_K8S_VERSION} \
  --control-plane-machine-count=1 \
  --worker-machine-count=1 \
  > capi-quickstart.yaml

clusterctl generate cluster capi-quickstart \
  --infrastructure azure \
  --kubernetes-version v1.36.1 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

# Cluster templates authenticate with Workload Identity by default. Modify the AzureClusterIdentity for ServicePrincipal authentication.
# See https://capz.sigs.k8s.io/topics/identities for more details.
yq -i "with(. | select(.kind == \"AzureClusterIdentity\"); .spec.type |= \"ServicePrincipal\" | .spec.clientSecret.name |= \"${AZURE_CLUSTER_IDENTITY_SECRET_NAME}\" | .spec.clientSecret.namespace |= \"${AZURE_CLUSTER_IDENTITY_SECRET_NAMESPACE}\")" capi-quickstart.yaml

clusterctl generate cluster capi-quickstart \
  --kubernetes-version v1.36.1 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

This creates a YAML file named capi-quickstart.yaml with a predefined list of Cluster API objects; Cluster, Machines, Machine Deployments, etc.

The file can be eventually modified using your editor of choice.

See clusterctl generate cluster for more details.

Apply the workload cluster

When ready, run the following command to apply the cluster manifest.

kubectl apply -f capi-quickstart.yaml

The output is similar to this:

cluster.cluster.x-k8s.io/capi-quickstart created
dockercluster.infrastructure.cluster.x-k8s.io/capi-quickstart created
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/capi-quickstart-control-plane created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-control-plane created
machinedeployment.cluster.x-k8s.io/capi-quickstart-md-0 created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/capi-quickstart-md-0 created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/capi-quickstart-md-0 created

Accessing the workload cluster

The cluster will now start provisioning. You can check status with:

kubectl get cluster

You can also get an “at glance” view of the cluster and its resources by running:

clusterctl describe cluster capi-quickstart

and see an output similar to this:

NAME              PHASE         AGE   VERSION
capi-quickstart   Provisioned   8s    v1.36.1

To verify the first control plane is up:

kubectl get kubeadmcontrolplane

You should see an output is similar to this:

NAME                    CLUSTER           INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE    VERSION
capi-quickstart-g2trk   capi-quickstart   true                                 3                  3         3             4m7s   v1.36.1

After the first control plane node is up and running, we can retrieve the workload cluster Kubeconfig.

clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

kind get kubeconfig --name capi-quickstart > capi-quickstart.kubeconfig

Install a Cloud Provider

The Kubernetes in-tree cloud provider implementations are being removed in favor of external cloud providers (also referred to as “out-of-tree”). This requires deploying a new component called the cloud-controller-manager which is responsible for running all the cloud specific controllers that were previously run in the kube-controller-manager. To learn more, see this blog post.

helm install --kubeconfig=./capi-quickstart.kubeconfig --repo https://raw.githubusercontent.com/kubernetes-sigs/cloud-provider-azure/master/helm/repo cloud-provider-azure --generate-name --set infra.clusterName=capi-quickstart --set cloudControllerManager.clusterCIDR="192.168.0.0/16"

cat > cloud.conf <<EOF
[Global]
auth-url=<your_auth_url>
application-credential-id=<your_credential_id>
application-credential-secret=<your_credential_secret>
region=<your_region>
domain-name=<your_domain_name>
EOF

kubectl --kubeconfig=./capi-quickstart.kubeconfig -n kube-system create secret generic cloud-config --from-file=cloud.conf

kubectl apply --kubeconfig=./capi-quickstart.kubeconfig -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-roles.yaml
kubectl apply --kubeconfig=./capi-quickstart.kubeconfig -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-role-bindings.yaml
kubectl apply --kubeconfig=./capi-quickstart.kubeconfig -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml

kubectl create secret -n kube-system generic capi-quickstart-oxide-cloud-controller-manager \
    --kubeconfig ./capi-quickstart.kubeconfig \
    --from-literal=oxide-host=${OXIDE_HOST} \
    --from-literal=oxide-token=${OXIDE_TOKEN} \
    --from-literal=oxide-project=${OXIDE_PROJECT}

helm upgrade --install capi-quickstart \
    oci://ghcr.io/oxidecomputer/helm-charts/oxide-cloud-controller-manager \
    --namespace kube-system \
    --kubeconfig ./capi-quickstart.kubeconfig \
    --wait

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: scaleway-secret
  namespace: kube-system
type: Opaque
stringData:
  SCW_ACCESS_KEY: "xxxxxxxxxxxxxxxx"
  SCW_SECRET_KEY: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
  SCW_DEFAULT_PROJECT_ID: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx"
  SCW_DEFAULT_REGION: "fr-par"
  SCW_DEFAULT_ZONE: "fr-par-1"
  PN_ID: "" # If your have a private network on your cluster, you may set its ID here.
EOF

kubectl apply -f https://raw.githubusercontent.com/scaleway/scaleway-cloud-controller-manager/master/examples/k8s-scaleway-ccm-latest.yml

Deploy a CNI solution

Calico is used here as an example.

helm repo add projectcalico https://docs.tigera.io/calico/charts --kubeconfig=./capi-quickstart.kubeconfig && \
helm install calico projectcalico/tigera-operator --kubeconfig=./capi-quickstart.kubeconfig -f https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-azure/main/templates/addons/calico/values.yaml --namespace tigera-operator --create-namespace

kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes

NAME                                          STATUS   ROLES           AGE    VERSION
capi-quickstart-vs89t-gmbld                   Ready    control-plane   5m33s  v1.36.1
capi-quickstart-vs89t-kf9l5                   Ready    control-plane   6m20s  v1.36.1
capi-quickstart-vs89t-t8cfn                   Ready    control-plane   7m10s  v1.36.1
capi-quickstart-md-0-55x6t-5649968bd7-8tq9v   Ready    <none>          6m5s   v1.36.1
capi-quickstart-md-0-55x6t-5649968bd7-glnjd   Ready    <none>          6m9s   v1.36.1
capi-quickstart-md-0-55x6t-5649968bd7-sfzp6   Ready    <none>          6m9s   v1.36.1

kubectl get vm

NAME                                  AGE    STATUS    READY
capi-quickstart-control-plane-7s945   167m   Running   True
capi-quickstart-md-0-zht5j            164m   Running   True

kubectl get vmi

NAME                                  AGE    PHASE     IP             NODENAME             READY
capi-quickstart-control-plane-7s945   167m   Running   10.244.82.16   kind-control-plane   True
capi-quickstart-md-0-zht5j            164m   Running   10.244.82.17   kind-control-plane   True

curl https://raw.githubusercontent.com/projectcalico/calico/v3.29.1/manifests/calico.yaml -o calico-workload.yaml

sed -i -E 's|^( +)# (- name: CALICO_IPV4POOL_CIDR)$|\1\2|g;'\
's|^( +)# (  value: )"192.168.0.0/16"|\1\2"10.243.0.0/16"|g;'\
'/- name: CLUSTER_TYPE/{ n; s/( +value: ").+/\1k8s"/g };'\
'/- name: CALICO_IPV4POOL_IPIP/{ n; s/value: "Always"/value: "Never"/ };'\
'/- name: CALICO_IPV4POOL_VXLAN/{ n; s/value: "Never"/value: "Always"/};'\
'/# Set Felix endpoint to host default action to ACCEPT./a\            - name: FELIX_VXLANPORT\n              value: "6789"' \
calico-workload.yaml

kubectl --kubeconfig=./capi-quickstart.kubeconfig create -f calico-workload.yaml

kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes

helm repo add projectcalico https://docs.tigera.io/calico/charts

helm install calico-crds projectcalico/crd.projectcalico.org.v1 \
  --kubeconfig=./capi-quickstart.kubeconfig

helm upgrade --install calico projectcalico/tigera-operator \
  --kubeconfig=./capi-quickstart.kubeconfig \
  --namespace tigera-operator \
  --create-namespace \
  --set 'installation.calicoNetwork.ipPools[0].encapsulation=VXLAN' \
  --set 'installation.calicoNetwork.ipPools[0].cidr=192.168.0.0/16' \
  --set 'installation.calicoNetwork.bgp=Disabled'

kubectl --kubeconfig=./capi-quickstart.kubeconfig \
  apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.26.1/manifests/calico.yaml

kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes

NAME                                          STATUS   ROLES           AGE    VERSION
capi-quickstart-vs89t-gmbld                   Ready    control-plane   5m33s  v1.36.1
capi-quickstart-vs89t-kf9l5                   Ready    control-plane   6m20s  v1.36.1
capi-quickstart-vs89t-t8cfn                   Ready    control-plane   7m10s  v1.36.1
capi-quickstart-md-0-55x6t-5649968bd7-8tq9v   Ready    <none>          6m5s   v1.36.1
capi-quickstart-md-0-55x6t-5649968bd7-glnjd   Ready    <none>          6m9s   v1.36.1
capi-quickstart-md-0-55x6t-5649968bd7-sfzp6   Ready    <none>          6m9s   v1.36.1

Clean Up

Delete workload cluster.

kubectl delete cluster capi-quickstart

Delete management cluster

kind delete cluster

Next steps

• Create a second workload cluster. Simply follow the steps outlined above, but remember to provide a different name for your second workload cluster.
• Deploy applications to your workload cluster. Use the CNI deployment steps for pointers.
• See the clusterctl documentation for more detail about clusterctl supported actions.

Cluster API Operator Quickstart

This section provides a quickstart guide for using the Cluster API Operator to create a Kubernetes cluster. To use the clusterctl quickstart path, visit this quickstart guide.

Quickstart

This is a quickstart guide for getting Cluster API Operator up and running on your Kubernetes cluster.

For more detailed information, please refer to the full documentation.

Prerequisites

• Running Kubernetes cluster.
• kubectl for interacting with the management cluster.
• Helm for installing operator on the cluster (optional).

Install and configure Cluster API Operator

Configuring credential for cloud providers

Instead of using environment variables as clusterctl does, Cluster API Operator uses Kubernetes secrets to store credentials for cloud providers. Refer to provider documentation on which credentials are required.

This example uses AWS provider, but the same approach can be used for other providers.

export CREDENTIALS_SECRET_NAME="credentials-secret"
export CREDENTIALS_SECRET_NAMESPACE="default"

kubectl create secret generic "${CREDENTIALS_SECRET_NAME}" --from-literal=AWS_B64ENCODED_CREDENTIALS="${AWS_B64ENCODED_CREDENTIALS}" --namespace "${CREDENTIALS_SECRET_NAMESPACE}"

Installing Cluster API Operator

Add CAPI Operator & cert manager helm repository:

helm repo add capi-operator https://kubernetes-sigs.github.io/cluster-api-operator
helm repo add jetstack https://charts.jetstack.io --force-update
helm repo update

Install cert manager:

helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set installCRDs=true

Deploy Cluster API components with docker provider using a single command during operator installation

helm install capi-operator capi-operator/cluster-api-operator --create-namespace -n capi-operator-system --set infrastructure.docker.enabled=true --set configSecret.name=${CREDENTIALS_SECRET_NAME} --set configSecret.namespace=${CREDENTIALS_SECRET_NAMESPACE}  --wait --timeout 90s

Docker provider can be replaced by any provider supported by clusterctl.

Other options for installing Cluster API Operator are described in full documentation.

Example API Usage

Deploy latest version of core Cluster API components:

apiVersion: operator.cluster.x-k8s.io/v1alpha2
kind: CoreProvider
metadata:
  name: cluster-api
  namespace: capi-system

Deploy Cluster API AWS provider with specific version, custom manager options and flags:

---
apiVersion: operator.cluster.x-k8s.io/v1alpha2
kind: InfrastructureProvider
metadata:
 name: aws
 namespace: capa-system
spec:
 version: v2.1.4
 configSecret:
   name: credentials-secret

Concepts

Similar to how you can use StatefulSets or Deployments in Kubernetes to manage a group of Pods, in Cluster API you can use custom resources like KubeadmControlPlane (a control plane implementation) to manage a set of control plane Machines, or you can use MachineDeployments to manage a group of worker Machines, each one of them representing a host server and the corresponding Kubernetes Node.

Extensibility is at the core of Cluster API and Cluster API providers like Cluster API provider VSphere, AWS, GCP etc. can be used to deploy Cluster API managed Clusters to your preferred infrastructure, as well as to configure many other parts of the system.

See also Quick start.

Management cluster

A Kubernetes cluster where Cluster API and one or more Cluster API providers run, and that can be used to manage the lifecycle of your Kubernetes Cluster via a set of custom resources such as Cluster or Machines.

Cluster

A “Cluster” is a custom resource that represent a Kubernetes cluster whose lifecycle is managed by Cluster API, usually also referred to as workload cluster.

Common properties such as network CIDRs are modeled as fields on the Cluster’s spec. Any information that is provider-specific is part of the custom resources referenced via infrastructureRef or controlPlaneRef and is not portable between different providers.

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: my-cluster
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 192.168.0.0/16
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: VSphereCluster
    name: my-cluster-infrastructure
  controlPlaneRef:
    apiGroup: controlplane.cluster.x-k8s.io
    kind: KubeadmControlPlane
    name: my-control-plane

In most recent versions of Cluster API, the Cluster object can be used as a single point of control for the entire cluster. See ClusterClass

Machine

A “Machine” is a custom resource providing the declarative spec for infrastructure hosting a Kubernetes Node (for example, a VM).

apiVersion: cluster.x-k8s.io/v1beta2
kind: Machine
metadata:
  name: my-machine
spec:
  clusterName: my-cluster
  version: v1.35.0
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: VSphereMachineTemplate
    name: my-machine-infrastructure
  bootstrap:
    configRef:
      apiGroup: bootstrap.cluster.x-k8s.io
      kind: KubeadmConfigTemplate
      name: my-bootstrap-config
status:
  nodeRef:
    name: the-node-running-on-my-machine

Common fields such as the Kubernetes version are modeled as fields on the Machine’s spec. Any information that is provider-specific is part of the custom resources referenced via infrastructureRef or bootstrap.configRef and is not portable between different providers.

If a new Machine object is created, a provider-specific controller will provision and install a new host to register as a new Node matching the Machine spec. If a Machine object is deleted, its underlying infrastructure and corresponding Node will be deleted.

Like for Pods in Kubernetes, also for Machines in Cluster API it is more convenient to not manage single Machines directly. Instead you should use resources like KubeadmControlPlane (a control plane implementation), MachineDeployments or MachinePools to manage a group of Machines.

Machine Immutability (In-place update vs. Replace)

From the perspective of Cluster API, all Machines are immutable: once they are created, they are never updated (except for labels, annotations and status), only deleted.

For this reason, MachineDeployments are preferable. MachineDeployments handle changes to machines by replacing them, in the same way core Deployments handle changes to Pod specifications.

Over time several improvement have been applied to Cluster API in oder to perform machine rollout only when necessary and for minimizing risks and impact of this operation on users workloads.

Starting from Cluster API v1.12, users can intentionally trade off some of the benefits that they get of Machine immutability by using Cluster API extensions points to add the capability to perform in-place updates under well-defined circumstances.

Notably, the Cluster API user experience will remain the same no matter of the in-place update feature is enabled or not, because ultimately users should care ONLY about the desired state.

Cluster API is responsible to choose the best strategy to achieve desired state, and with the introduction of update extensions, Cluster API is expanding the set of tools that can be used to achieve the desired state.

Infrastructure provider

A component responsible for the provisioning of infrastructure/computational resources required by the Cluster or by Machines (e.g. VMs, networking, etc.). For example, cloud Infrastructure Providers include AWS, Azure, and Google, and bare metal Infrastructure Providers include VMware, MAAS, and metal3.io.

When there is more than one way to obtain resources from the same Infrastructure Provider (such as AWS offering both EC2 and EKS), each way is referred to as a variant.

Control plane provider

A component responsible for the provisioning and for the management of the control plane of your Kubernetes Cluster, like e.g. the KubeadmControlPlane provider.

Control plane providers can take different approach on how to manage the control plane;

• Self-provisioned: A Kubernetes control plane consisting of pods or machines wholly managed by a single Cluster API deployment. e.g kubeadm uses static pods for running components such as kube-apiserver, kube-controller-manager and kube-scheduler on control plane machines.

• Pod-based deployments require an external hosting cluster. The control plane components are deployed using standard Deployment and StatefulSet objects and the API is exposed using a Service.

• External or Managed control planes are offered and controlled by some system other than Cluster API, such as GKE, AKS, EKS, or IKS.

Bootstrap provider

A component responsible for turning a server into a Kubernetes node as well as for:

1. Generating the cluster certificates, if not otherwise specified
2. Initializing the control plane, and gating the creation of other nodes until it is complete
3. Joining control plane and worker nodes to the cluster

Boostrap provider achieve this goal by generating BootstrapData, which contains the Machine or Node role-specific initialization data (usually cloud-init). The bootstrap data is used by the Infrastructure Provider to bootstrap a Machine into a Node.

KubeadmControlPlane

The KubeadmControlPlane is a custom resource that is provided by the Kubeadm provider, and that allows to manage a set of Machines hosting control plane Nodes created with kubeadm.

Other control plane providers implement similar resources as well.

MachineDeployment

A MachineDeployment provides declarative updates for Machines and MachineSets.

A MachineDeployment works similarly to a core Kubernetes Deployment. A MachineDeployment reconciles changes to a Machine spec by rolling out changes to 2 MachineSets, the old and the newly updated.

MachinePool

A MachinePool is a declarative spec for a group of Machines. It is similar to a MachineDeployment, but is specific to a particular Infrastructure Provider. For more information, please check out MachinePool.

MachineSet

A MachineSet’s purpose is to maintain a stable set of Machines running at any given time.

A MachineSet works similarly to a core Kubernetes ReplicaSet. MachineSets are not meant to be used directly, but are the mechanism MachineDeployments use to reconcile desired state.

MachineHealthCheck

A MachineHealthCheck defines the conditions when a Node should be considered missing or unhealthy.

If the Node matches these unhealthy conditions for a given user-configured time, the MachineHealthCheck initiates remediation of the Node. Remediation of Nodes is performed by replacing the corresponding Machine.

MachineHealthChecks will only remediate Nodes if they are owned by a MachineSet. This ensures that the Kubernetes cluster does not lose capacity, since the MachineSet will create a new Machine to replace the failed Machine.

Custom Resource Definitions (CRDs)

A CustomResourceDefinition is a built-in resource that lets you extend the Kubernetes API. Each CustomResourceDefinition represents a customization of a Kubernetes installation. The Cluster API provides and relies on several CustomResourceDefinitions:

Cluster API Manifesto

Intro

Taking inspiration from Tim Hockin’s talk at KubeCon NA 2023, also for the Cluster API project is important to define the long term vision, the manifesto of “where we are going” and “why”.

This document would hopefully provide valuable context for all users, contributors and companies investing in this project, as well as act as compass for all reviewers and maintainers currently working on it.

Community

Together we can go far.

The Cluster API community is the foundation for this project’s past, present and future. The project will continue to encourage and praise active participation and contribution.

We are an active part of a bigger ecosystem

The Cluster API community is an active part of Kubernetes SIG Cluster Lifecycle, of the broader Kubernetes community and of the CNCF.

CNCF provides the core values this project recognizes and contributes to. The Kubernetes community provides most of the practices and policies this project abides to or is inspired by.

Core goals and design principles

The project remains true to its original goals and design principles:

Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters.

Nowadays, like at the beginning of the project, some concepts from the above statement deserve further clarification.

Declarative APIs

The Cluster API project motto is “Kubernetes all the way down”, and this boils down to two elements.

The target state of a cluster can be defined using Kubernetes declarative APIs.

The project also implements controllers – Kubernetes reconcile loops – ensuring that desired and current state of the cluster will remain consistent over time.

The combination of those elements, declarative APIs and controllers, defines “how” this project aims to make Kubernetes and Cluster API a stable, reliable and consistent platform that just works to enable higher order business value supported by cloud-native applications.

Simplicity

Kubernetes Cluster lifecycle management is a complex problem space, especially if you consider doing this across so many different types of infrastructures.

Hiding this complexity behind a simple declarative API is “why” the Cluster API project ultimately exists.

The project is strongly committed to continue its quest in defining a set of common API primitives working consistently across all infrastructures (one API to rule them all).

Working towards graduating our API to v1 will be the next step in this journey.

While doing so, the project should be inspired by Tim Hockin’s talk, and continue to move forward without increasing operational and conceptual complexity for Cluster API’s users.

The right to be Unfinished

Like Kubernetes, also the Cluster API project claims the right to remain unfinished, because there is still a strong, foundational need to continuously evolve, improve and adapt to the changing needs of Cluster API’s users and to the growing Cloud Native ecosystem.

What is important to notice, is that being a project that is “continuously evolving” is not in contrast with another request from Cluster API’s users, which is about the project being stable, as expected by a system that has “crossed the chasm”.

Those two requests from Cluster API’s users are two sides of the same coin, a reminder that Cluster API must “evolve responsibly” by ensuring upgrade paths and avoiding (or at least minimizing) disruptions for users.

The Cluster API project will continue to “evolve responsibly” by abiding to the same guarantees that Kubernetes offers for its own API resources, and by cleanly defining the guarantees offered for the public go modules and packages.

Also ensuring a continuous and obsessive focus on CI signal, test coverage and test flakes, a predictable release calendar, and clear documentation of the release support policies and of the compatibility matrix for each release are part of the continuous effort to “evolve responsibly”.

The complexity budget

Tim Hockins explains the idea of complexity budget very well in his talk:

There is a finite amount of complexity that a project can absorb over a certain amount of time; when the complexity budget runs out, bad things happen, quality decreases, we can’t fix bugs timely etc.

Since the beginning of the Cluster API project, its maintainers intuitively handled the complexity budget by following this approach:

“We’ve got to say no to things today, so we can afford to do interesting things tomorrow”.

This is something that is never done lightly, and it is always the result of an open discussion considering the status of the codebase, the status of the project CI signal, the complexity of the new feature etc. .

Being very pragmatic, also the resources committed to implement and to maintain a feature over time must be considered when doing such an evaluation, because a model where everything falls on the shoulders of a small set of core maintainers is not sustainable.

On the other side of this coin, Cluster API maintainer’s also claim the right to reconsider new ideas or ideas previously put on hold whenever there are the conditions and the required community consensus to work on it.

Probably the most well-known case of this is about Cluster API maintainers repeatedly deferring on change requests about nodes mutability in the initial phases of the project, while starting to embrace some mutable behavior in recent releases.

Core and providers

Together we can go far.

The Cluster API project is committed to keep working with the broader CAPI community – all the Cluster API providers – as a single team in order to continuously improve and expand the capability of this solution.

As we learned the hard way, the extensibility model implemented by CAPI to support so many providers requires a complementary effort to continuously explore new ways to offer a cohesive solution, not a bag of parts.

It is important to continue and renew efforts to make it easier to bootstrap and operate a system composed of many components, to ensure consistent APIs and behaviors, to ensure quality across the board.

This effort lays its foundation in all the provider maintainers being committed to this goal, while the Cluster API project will be the venue where common guidelines are discussed and documented, as well as the place of choice where common components or utilities are developed and hosted.

Cluster Management Tasks

This section provides details for some of the operations that need to be performed when managing clusters.

Certificate Management

This section details some tasks related to certificate management.

Using Custom Certificates

Using Custom Certificates

Cluster API expects certificates and keys used for bootstrapping to follow the below convention. CABPK generates new certificates using this convention if they do not already exist.

Each certificate must be stored in a single secret named one of:

The certificates must also be labeled with the key-value pair cluster.x-k8s.io/cluster-name=[cluster name] (where [cluster name] is the name of the cluster it should be used with).

Example

apiVersion: v1
kind: Secret
metadata:
  name: cluster1-ca
  labels:
    cluster.x-k8s.io/cluster-name: cluster1
type: kubernetes.io/tls
data:
  tls.crt: <base 64 encoded PEM>
  tls.key: <base 64 encoded PEM>

Generating a Kubeconfig

Generating a Kubeconfig with your own CA

This guide applies when you are using custom certificates for a Cluster API workload cluster, rather than relying on automatically generated certificates.

1. Create a new Certificate Signing Request (CSR) for the admin user with the system:masters Kubernetes role, or specify any other role under O.

   openssl req  -subj "/CN=admin/O=system:masters" -new -newkey rsa:2048 -nodes -keyout admin.key  -out admin.csr
   

2. Sign the CSR using the [cluster-name]-ca key:

   openssl x509 -req -in admin.csr -CA tls.crt -CAkey tls.key -CAcreateserial -out admin.crt -days 5 -sha256
   

3. Update your kubeconfig with the sign key:

   kubectl config set-credentials cluster-admin --client-certificate=admin.crt --client-key=admin.key --embed-certs=true
   

Auto Rotate Certificates in KCP

Automatically rotating certificates using Kubeadm Control Plane provider

When using Kubeadm Control Plane provider (KCP) it is possible to configure automatic certificate rotations. KCP does this by triggering a rollout when the certificates on the control plane machines are about to expire.

If configured, the certificate rollout feature is available for all new and existing control plane machines.

Configuring Machine Rollout

To configure a rollout on the KCP machines you need to set .rolloutBefore.certificatesExpiryDays (minimum of 7 days).

Example:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlane
metadata:
  name: example-control-plane
spec:
  rollout:
    before:
      certificatesExpiryDays: 21 # trigger a rollout if certificates expire within 21 days
  kubeadmConfigSpec:
    clusterConfiguration:
      ...
    initConfiguration:
      ...
    joinConfiguration:
      ...
  machineTemplate:
    spec:
      infrastructureRef:
        ...
  replicas: 1
  version: v1.23.3

It is strongly recommended to set the certificatesExpiryDays to a large enough value so that all the machines will have time to complete rollout well in advance before the certificates expire.

Triggering Machine Rollout for Certificate Expiry

KCP uses the value in the corresponding Control Plane machine’s Machine.Status.CertificatesExpiryDate to check if a machine’s certificates are going to expire and if it needs to be rolled out.

Machine.Status.CertificatesExpiryDate gets its value from one of the following 2 places:

• machine.cluster.x-k8s.io/certificates-expiry annotation value on the Machine object. This annotation is not applied by default and it can be set by users to manually override the certificate expiry information.
• machine.cluster.x-k8s.io/certificates-expiry annotation value on the Bootstrap Config object referenced by the machine. This value is automatically set for machines bootstrapped with CABPK that are owned by the KCP resource.

The annotation value is a RFC3339 format timestamp. The annotation value on the machine object, if provided, will take precedence.

Bootstrap

This section provides details about bootstrap providers.

Cluster API bootstrap provider kubeadm

What is the Cluster API bootstrap provider kubeadm?

Cluster API bootstrap provider Kubeadm (CABPK) is a component responsible for generating a cloud-init script to turn a Machine into a Kubernetes Node. This implementation uses kubeadm for Kubernetes bootstrap.

Resources

• design doc
• The Kubebuilder Book

How does CABPK work?

Assuming you have deployed the CAPI and CAPD controllers, create a Cluster object and its corresponding DockerCluster infrastructure object.

kind: DockerCluster
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
metadata:
  name: my-cluster-docker
---
kind: Cluster
apiVersion: cluster.x-k8s.io/v1beta2
metadata:
  name: my-cluster
spec:
  infrastructureRef:
    kind: DockerCluster
    apiGroup: infrastructure.cluster.x-k8s.io
    name: my-cluster-docker

Now you can start creating machines by defining a Machine, its corresponding DockerMachine object, and the KubeadmConfig bootstrap object.

kind: KubeadmConfig
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
metadata:
  name: my-control-plane1-config
---
kind: DockerMachine
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
metadata:
  name: my-control-plane1-docker
---
kind: Machine
apiVersion: cluster.x-k8s.io/v1beta2
metadata:
  name: my-control-plane1
  labels:
    cluster.x-k8s.io/cluster-name: my-cluster
    cluster.x-k8s.io/control-plane: "true"
    set: controlplane
spec:
  bootstrap:
    configRef:
      apiGroup: bootstrap.cluster.x-k8s.io
      kind: KubeadmConfig
      name: my-control-plane1-config
  infrastructureRef:
    apiGroup: infrastructure.cluster.x-k8s.io
    kind: DockerMachine
    name: my-control-plane1-docker
  version: "v1.19.1"

CABPK’s main responsibility is to convert a KubeadmConfig bootstrap object into a cloud-init script that is going to turn a Machine into a Kubernetes Node using kubeadm.

The cloud-init script will be saved into a secret KubeadmConfig.Status.DataSecretName and then the infrastructure provider (CAPD in this example) will pick up this value and proceed with the machine creation and the actual bootstrap.

KubeadmConfig objects

The KubeadmConfig object allows full control of Kubeadm init/join operations by exposing raw InitConfiguration, ClusterConfiguration and JoinConfiguration objects.

InitConfiguration and JoinConfiguration exposes Patches field which can be used to specify the patches from a directory, this support is available from K8s 1.22 version onwards.

CABPK will fill in some values if they are left empty with sensible defaults:

IMPORTANT! overriding above defaults could lead to broken Clusters.

[1] if both clusterConfiguration.KubernetesVersion and Machine.Spec.Version are empty, the latest Kubernetes version will be installed (as defined by the default kubeadm behavior).

Examples

Valid combinations of configuration objects are:

• for KCP, InitConfiguration and ClusterConfiguration for the first control plane node; JoinConfiguration for additional control plane nodes
• for machine deployments, JoinConfiguration for worker nodes

Bootstrap control plane node:

kind: KubeadmConfig
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
metadata:
  name: my-control-plane1-config

Additional control plane nodes:

kind: KubeadmConfig
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
metadata:
  name: my-control-plane2-config
spec:
  joinConfiguration:
    controlPlane: {}

worker nodes:

kind: KubeadmConfig
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
metadata:
  name: my-worker1-config

Bootstrap Orchestration

CABPK supports multiple control plane machines initing at the same time. The generation of cloud-init scripts of different machines is orchestrated in order to ensure a cluster bootstrap process that will be compliant with the correct Kubeadm init/join sequence. More in detail:

1. cloud-config-data generation starts only after Cluster.Status.InfrastructureReady flag is set to true.
2. at this stage, cloud-config-data will be generated for the first control plane machine only, keeping on hold additional control plane machines existing in the cluster, if any (kubeadm init).
3. after the ControlPlaneInitialized conditions on the cluster object is set to true, the cloud-config-data for all the other machines are generated (kubeadm join/join —control-plane).

Certificate Management

The user can choose two approaches for certificate management:

1. provide required certificate authorities (CAs) to use for kubeadm init/kubeadm join --control-plane; such CAs should be provided as a Secrets objects in the management cluster.
2. let KCP to generate the necessary Secrets objects with a self-signed certificate authority for kubeadm

See here for more info about certificate management with kubeadm.

Additional Features

The KubeadmConfig object supports customizing the content of the config-data. The following examples illustrate how to specify these options. They should be adapted to fit your environment and use case.

• KubeadmConfig.Files specifies additional files to be created on the machine, either with content inline or by referencing a secret.

  files:
  - contentFrom:
      secret:
        key: node-cloud.json
        name: ${CLUSTER_NAME}-md-0-cloud-json
    owner: root:root
    path: /etc/kubernetes/cloud.json
    permissions: "0644"
  - path: /etc/kubernetes/cloud.json
    owner: "root:root"
    permissions: "0644"
    content: |
      {
        "cloud": "CustomCloud"
      }
  

• KubeadmConfig.BootCommands specifies a list of commands to be executed very early in the boot process

  bootCommands:
    - cloud-init-per once mymkfs mkfs /dev/vdb
  

• KubeadmConfig.PreKubeadmCommands specifies a list of commands to be executed before kubeadm init/join

  preKubeadmCommands:
    - hostname "{{ ds.meta_data.hostname }}"
    - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
  

• KubeadmConfig.PostKubeadmCommands same as above, but after kubeadm init/join

  postKubeadmCommands:
    - echo "success" >/var/log/my-custom-file.log
  

• KubeadmConfig.Users specifies a list of users to be created on the machine

  users:
    - name: capiuser
      sshAuthorizedKeys:
      - '${SSH_AUTHORIZED_KEY}'
      sudo: ALL=(ALL) NOPASSWD:ALL
  

• KubeadmConfig.NTP specifies NTP settings for the machine

  ntp:
    servers:
      - IP_ADDRESS
    enabled: true
  

• KubeadmConfig.DiskSetup specifies options for the creation of partition tables and file systems on devices.

  diskSetup:
    filesystems:
    - device: /dev/disk/azure/scsi1/lun0
      extraOpts:
      - -E
      - lazy_itable_init=1,lazy_journal_init=1
      filesystem: ext4
      label: etcd_disk
    - device: ephemeral0.1
      filesystem: ext4
      label: ephemeral0
      replaceFS: ntfs
    partitions:
    - device: /dev/disk/azure/scsi1/lun0
      layout: true
      overwrite: false
      tableType: gpt
  

• KubeadmConfig.Mounts specifies a list of mount points to be setup.

  mounts:
  - - LABEL=etcd_disk
    - /var/lib/etcddisk
  

• KubeadmConfig.Verbosity specifies the kubeadm log level verbosity

  verbosity: 10
  

For more information on cloud-init options, see cloud config examples.

Kubelet Configuration

CAPBK has several ways to configure kubelet.

• Kubelet Configuration
  • Pass KubeletConfiguration file via KubeadmConfigSpec.files
    • KubeadmControlPlaneTemplate
    • KubeadmConfigTemplate
  • Set kubelet flags via KubeadmConfigSpec.kubeletExtraArgs
    • KubeadmControlPlaneTemplate
    • KubeadmConfigTemplate
  • Use kubeadm’s kubeletconfiguration patch target
    • KubeadmControlPlaneTemplate
    • KubeadmConfigTemplate

Pass KubeletConfiguration file via KubeadmConfigSpec.files

You can use KubeadmConfigSpec.files to put any files on nodes. This example puts a KubeletConfiguration file on nodes via KubeadmConfigSpec.files, and makes kubelet use it via KubeadmConfigSpec.kubeletExtraArgs. You can check available configurations of KubeletConfiguration on Kubelet Configuration (v1beta1) | Kubernetes.

This method is easy to replace the whole kubelet configuration generated by kubeadm, but it is not easy to replace only a part of the kubelet configuration.

KubeadmControlPlaneTemplate

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlaneTemplate
metadata:
  name: cloudinit-control-plane
  namespace: default
spec:
  template:
    spec:
      kubeadmConfigSpec:
        files:
        # We put a KubeletConfiguration file on nodes via KubeadmConfigSpec.files
        # In this example, we directly put the file content in the KubeadmConfigSpec.files.content field.
        - path: /etc/kubernetes/kubelet/config.yaml
          owner: "root:root"
          permissions: "0644"
          content: |
            apiVersion: kubelet.config.k8s.io/v1beta1
            kind: KubeletConfiguration
            kubeReserved:
              cpu: "1"
              memory: "2Gi"
              ephemeral-storage: "1Gi"
            systemReserved:
              cpu: "500m"
              memory: "1Gi"
              ephemeral-storage: "1Gi"
            evictionHard:
              memory.available: "500Mi"
              nodefs.available: "10%"
            authentication:
              anonymous:
                enabled: false
              webhook:
                cacheTTL: 0s
                enabled: true
              x509:
                clientCAFile: /etc/kubernetes/pki/ca.crt
            authorization:
              mode: Webhook
              webhook:
                cacheAuthorizedTTL: 0s
                cacheUnauthorizedTTL: 0s
            cgroupDriver: systemd
            clusterDNS:
            - 10.128.0.10
            clusterDomain: cluster.local
            containerRuntimeEndpoint: ""
            cpuManagerReconcilePeriod: 0s
            evictionPressureTransitionPeriod: 0s
            fileCheckFrequency: 0s
            healthzBindAddress: 127.0.0.1
            healthzPort: 10248
            httpCheckFrequency: 0s
            imageMinimumGCAge: 0s
            logging:
              flushFrequency: 0
              options:
                json:
                  infoBufferSize: "0"
              verbosity: 0
            memorySwap: {}
            nodeStatusReportFrequency: 0s
            nodeStatusUpdateFrequency: 0s
            rotateCertificates: true
            runtimeRequestTimeout: 0s
            shutdownGracePeriod: 0s
            shutdownGracePeriodCriticalPods: 0s
            staticPodPath: /etc/kubernetes/manifests
            streamingConnectionIdleTimeout: 0s
            syncFrequency: 0s
            volumeStatsAggPeriod: 0s
        initConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
            # Here we configure kubelet to use the KubeletConfiguration file we put on nodes via KubeadmConfigSpec.files
            kubeletExtraArgs:
              - name: config
                value: "/etc/kubernetes/kubelet/config.yaml"
        joinConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
            # Here we configure kubelet to use the KubeletConfiguration file we put on nodes via KubeadmConfigSpec.files
            kubeletExtraArgs:
              - name: config
                value: "/etc/kubernetes/kubelet/config.yaml"

KubeadmConfigTemplate

apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
metadata:
  name: cloudinit-default-worker-bootstraptemplate
  namespace: default
spec:
  template:
    spec:
      files:
      # We puts a KubeletConfiguration file on nodes via KubeadmConfigSpec.files
      # In this example, we directly put the file content in the KubeadmConfigSpec.files.content field.
      - path: /etc/kubernetes/kubelet/config.yaml
        owner: "root:root"
        permissions: "0644"
        content: |
          apiVersion: kubelet.config.k8s.io/v1beta1
          kind: KubeletConfiguration
          kubeReserved:
            cpu: "1"
            memory: "2Gi"
            ephemeral-storage: "1Gi"
          systemReserved:
            cpu: "500m"
            memory: "1Gi"
            ephemeral-storage: "1Gi"
          evictionHard:
            memory.available: "500Mi"
            nodefs.available: "10%"
          authentication:
            anonymous:
              enabled: false
            webhook:
              cacheTTL: 0s
              enabled: true
            x509:
              clientCAFile: /etc/kubernetes/pki/ca.crt
          authorization:
            mode: Webhook
            webhook:
              cacheAuthorizedTTL: 0s
              cacheUnauthorizedTTL: 0s
          cgroupDriver: systemd
          clusterDNS:
          - 10.128.0.10
          clusterDomain: cluster.local
          containerRuntimeEndpoint: ""
          cpuManagerReconcilePeriod: 0s
          evictionPressureTransitionPeriod: 0s
          fileCheckFrequency: 0s
          healthzBindAddress: 127.0.0.1
          healthzPort: 10248
          httpCheckFrequency: 0s
          imageMinimumGCAge: 0s
          logging:
            flushFrequency: 0
            options:
              json:
                infoBufferSize: "0"
            verbosity: 0
          memorySwap: {}
          nodeStatusReportFrequency: 0s
          nodeStatusUpdateFrequency: 0s
          rotateCertificates: true
          runtimeRequestTimeout: 0s
          shutdownGracePeriod: 0s
          shutdownGracePeriodCriticalPods: 0s
          staticPodPath: /etc/kubernetes/manifests
          streamingConnectionIdleTimeout: 0s
          syncFrequency: 0s
          volumeStatsAggPeriod: 0s
      joinConfiguration:
        nodeRegistration:
          criSocket: unix:///var/run/containerd/containerd.sock
          # Here we configure kubelet to use the KubeletConfiguration file we put on nodes via KubeadmConfigSpec.files
          kubeletExtraArgs:
            - name: config
              value: "/etc/kubernetes/kubelet/config.yaml"

Set kubelet flags via KubeadmConfigSpec.kubeletExtraArgs

We can pass kubelet command-line flags via KubeadmConfigSpec.kubeletExtraArgs. This example is equivalent to setting --kube-reserved, --system-reserved, and --eviction-hard flags for the kubelet command.

This method is useful when you want to set kubelet flags that are not configurable via the KubeletConfiguration file, however, it is not recommended to use this method to set flags that are configurable via the KubeletConfiguration file.

KubeadmControlPlaneTemplate

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlaneTemplate
metadata:
  name: kubelet-extra-args-control-plane
  namespace: default
spec:
  template:
    spec:
      kubeadmConfigSpec:
        initConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
            # Set kubelet flags via KubeadmConfigSpec.kubeletExtraArgs
            kubeletExtraArgs:
              - name: kube-reserved
                value: "cpu=1,memory=2Gi,ephemeral-storage=1Gi"
              - name: system-reserved
                value: "cpu=500m,memory=1Gi,ephemeral-storage=1Gi"
              - name: eviction-hard
                value: "memory.available<500Mi,nodefs.available<10%"
        joinConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
            # Set kubelet flags via KubeadmConfigSpec.kubeletExtraArgs
            kubeletExtraArgs:
              - name: kube-reserved
                value: "cpu=1,memory=2Gi,ephemeral-storage=1Gi"
              - name: system-reserved
                value: "cpu=500m,memory=1Gi,ephemeral-storage=1Gi"
              - name: eviction-hard
                value: "memory.available<500Mi,nodefs.available<10%"

KubeadmConfigTemplate

apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
metadata:
  name: kubelet-extra-args-default-worker-bootstraptemplate
  namespace: default
spec:
  template:
    spec:
      joinConfiguration:
        nodeRegistration:
          criSocket: unix:///var/run/containerd/containerd.sock
          # Set kubelet flags via KubeadmConfigSpec.kubeletExtraArgs
          kubeletExtraArgs:
            - name: kube-reserved
              value: "cpu=1,memory=2Gi,ephemeral-storage=1Gi"
            - name: system-reserved
              value: "cpu=500m,memory=1Gi,ephemeral-storage=1Gi"
            - name: eviction-hard
              value: "memory.available<500Mi,nodefs.available<10%"

Use kubeadm’s kubeletconfiguration patch target

We can use kubeadm’s kubeletconfiguration patch target to patch the kubelet configuration file. In this example, we put a patch file for kubeletconfiguration target in strategic patchtype on nodes via KubeadmConfigSpec.files. For more details, see Customizing components with the kubeadm API | Kubernetes

This method is useful when you want to change the kubelet configuration file partially on specific nodes. For example, you can deploy a partially patched kubelet configuration file on specific nodes based on the default configuration used for kubeadm init or kubeadm join.

KubeadmControlPlaneTemplate

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlaneTemplate
metadata:
  name: kubeadm-config-template-control-plane
  namespace: default
spec:
  template:
    spec:
      kubeadmConfigSpec:
        files:
        # Here we put a patch file for kubeletconfiguration target in strategic patchtype on nodes via KubeadmConfigSpec.files
        # The naming convention of the patch file is kubeletconfiguration{suffix}+{patchtype}.json where {suffix} is an string and {patchtype} is one of the following: strategic, merge, json.
        # {suffix} determines the order of the patch files. The patches are applied in the alpha-numerical order of the {suffix}.
        - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
          owner: "root:root"
          permissions: "0644"
          content: |
            {
              "apiVersion": "kubelet.config.k8s.io/v1beta1",
              "kind": "KubeletConfiguration",
              "kubeReserved": {
                "cpu": "1",
                "memory": "2Gi",
                "ephemeral-storage": "1Gi",
              },
              "systemReserved": {
                "cpu": "500m",
                "memory": "1Gi",
                "ephemeral-storage": "1Gi",
              },
              "evictionHard": {
                "memory.available": "500Mi",
                "nodefs.available": "10%",
              },
            }
        initConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
          # Here we specify the directory that contains the patch files
          patches:
            directory: /etc/kubernetes/patches
        joinConfiguration:
          nodeRegistration:
            criSocket: unix:///var/run/containerd/containerd.sock
          # Here we specify the directory that contains the patch files
          patches:
            directory: /etc/kubernetes/patches

KubeadmConfigTemplate

apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
metadata:
  name: kubeadm-config-template-default-worker-bootstraptemplate
  namespace: default
spec:
  template:
    spec:
      files:
      # Here we put a patch file for kubeletconfiguration target in strategic patchtype on nodes via KubeadmConfigSpec.files
      # The naming convention of the patch file is kubeletconfiguration{suffix}+{patchtype}.json where {suffix} is an string and {patchtype} is one of the following: strategic, merge, json.
      # {suffix} determines the order of the patch files. The patches are applied in the alpha-numerical order of the {suffix}.
      - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json
        owner: "root:root"
        permissions: "0644"
        content: |
          {
            "apiVersion": "kubelet.config.k8s.io/v1beta1",
            "kind": "KubeletConfiguration",
            "kubeReserved": {
              "cpu": "1",
              "memory": "2Gi",
              "ephemeral-storage": "1Gi",
            },
            "systemReserved": {
              "cpu": "500m",
              "memory": "1Gi",
              "ephemeral-storage": "1Gi",
            },
            "evictionHard": {
              "memory.available": "500Mi",
              "nodefs.available": "10%",
            },
          }
      joinConfiguration:
        nodeRegistration:
          criSocket: unix:///var/run/containerd/containerd.sock
        # Here we specify the directory that contains the patch files
        patches:
          directory: /etc/kubernetes/patches

Cluster API bootstrap provider MicroK8s

What is the Cluster API bootstrap provider MicroK8s?

Cluster API bootstrap provider MicroK8s (CABPM) is a component responsible for generating a cloud-init script to turn a Machine into a Kubernetes Node. This implementation uses MicroK8s for Kubernetes bootstrap.

Resources

• CABPM Repository
• Official MicroK8s site

CABPM configuration options

MicroK8s defines a MicroK8sControlPlane definition as well as the MachineDeployment to configure the control plane and worker nodes respectively. The MicroK8sControlPlane is linked in the cluster definition as shown in the following example:

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
spec:
  controlPlaneRef:
    apiGroup: controlplane.cluster.x-k8s.io
    kind: MicroK8sControlPlane
    name: capi-aws-control-plane

A control plane manifest section includes the Kubernetes version, the replica number as well as the MicroK8sConfig:

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: MicroK8sControlPlane
spec:
  controlPlaneConfig:
    initConfiguration:
      addons:
      - dns
      - ingress
  replicas: 3
  version: v1.23.0
  ......

The worker nodes are configured through the MachineDeployment object:

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  name: capi-aws-md-0
  namespace: default
spec:
  clusterName: capi-aws
  replicas: 2
  selector:
    matchLabels: null
  template:
    spec:
      clusterName: capi-aws
      version: v1.23.0     
      bootstrap:
        configRef:
          apiGroup: bootstrap.cluster.x-k8s.io
          kind: MicroK8sConfigTemplate
          name: capi-aws-md-0
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: MicroK8sConfigTemplate
metadata:
  name: capi-aws-md-0
  namespace: default
spec:
  template:
    spec: {}
......

In both the MicroK8sControlPlane and MicroK8sConfigTemplate you can set a MicroK8sConfig object. In the MicroK8sControlPlane case MicroK8sConfig is under MicroK8sConfig.spec.controlPlaneConfig whereas in MicroK8sConfigTemplate it is under MicroK8sConfigTemplate.spec.template.spec.

Some of the configuration options available via MicroK8sConfig are:

• MicroK8sConfig.spec.initConfiguration.joinTokenTTLInSecs: the time-to-live (TTL) of the token used to join nodes, defaults to 10 years.
• MicroK8sConfig.spec.initConfiguration.httpsProxy: the https proxy to be used, defaults to none.
• MicroK8sConfig.spec.initConfiguration.httpProxy: the http proxy to be used, defaults to none.
• MicroK8sConfig.spec.initConfiguration.noProxy: the no-proxy to be used, defaults to none.
• MicroK8sConfig.spec.initConfiguration.addons: the list of addons to be enabled, defaults to dns.
• MicroK8sConfig.spec.clusterConfiguration.portCompatibilityRemap: option to reuse the security group ports set for kubeadm, defaults to true.

How does CABPM work?

The main purpose of the MicroK8s bootstrap provider is to translate the users needs to a number of cloud-init files applicable for each type of cluster nodes. There are three types of cloud-inits:

• The first node cloud-init. That node will be a control plane node and will be the one where the addons are enabled.
• The control plane node cloud-init. The control plane nodes need to join a cluster and contribute to its HA.
• The worker node cloud-init. These nodes join the cluster as workers.

The cloud-init scripts are saved as secrets that then the infrastructure provider uses during the machine creation. For more information on cloud-init options, see cloud config examples.

Upgrading management and workload clusters

Considerations

Supported versions of Kubernetes

If you are upgrading the version of Kubernetes for a cluster managed by Cluster API, check that the running version of Cluster API on the Management Cluster supports the target Kubernetes version.

You may need to upgrade the version of Cluster API in order to support the target Kubernetes version.

In addition, you must always upgrade between Kubernetes minor versions in sequence, e.g. if you need to upgrade from Kubernetes v1.17 to v1.19, you must first upgrade to v1.18.

Images

For kubeadm based clusters, infrastructure providers require a “machine image” containing pre-installed, matching versions of kubeadm and kubelet, ensure that relevant infrastructure machine templates reference the appropriate image for the Kubernetes version.

Upgrading using Cluster API

The high level steps to fully upgrading a cluster are to first upgrade the control plane and then upgrade the worker machines.

Upgrading the control plane machines

How to upgrade the underlying machine image

To upgrade the control plane machines underlying machine images, the MachineTemplate resource referenced by the KubeadmControlPlane must be changed. Since MachineTemplate resources are immutable, the recommended approach is to

1. Copy the existing MachineTemplate.
2. Modify the values that need changing, such as instance type or image ID.
3. Create the new MachineTemplate on the management cluster.
4. Modify the existing KubeadmControlPlane resource to reference the new MachineTemplate resource in the infrastructureRef field.

The next step will trigger a rolling update of the control plane using the new values found in the new MachineTemplate.

How to upgrade the Kubernetes control plane version

To upgrade the Kubernetes control plane version make a modification to the KubeadmControlPlane resource’s Spec.Version field. This will trigger a rolling upgrade of the control plane and, depending on the provider, also upgrade the underlying machine image.

Some infrastructure providers, such as AWS, require that if a specific machine image is specified, it has to match the Kubernetes version specified in the KubeadmControlPlane spec. In order to only trigger a single upgrade, the new MachineTemplate should be created first and then both the Version and InfrastructureTemplate should be modified in a single transaction.

How to schedule a machine rollout

The KubeadmControlPlane and MachineDepoyment resources have a spec.rollout.after field that can be set to a timestamp (RFC-3339) after which a rollout should be triggered regardless of whether there were any changes to KubeadmControlPlane.spec/MachineDeployment.spec.template or not. This would roll out replacement nodes which can be useful e.g. to perform certificate rotation, reflect changes to machine templates, move to new machines, etc.

Note that this field can only be used for triggering a rollout, not for delaying one. Specifically, a rollout can also happen before the time specified in spec.rollout.after if any changes are made to the spec before that time.

The rollout can be triggered by running the following command:

# Trigger a KubeadmControlPlane rollout.
clusterctl alpha rollout restart kubeadmcontrolplane/my-kcp

# Trigger a MachineDeployment rollout.
clusterctl alpha rollout restart machinedeployment/my-md-0

Upgrading machines managed by a MachineDeployment

Upgrades are not limited to just the control plane. This section is not related to Kubeadm control plane specifically, but is the final step in fully upgrading a Cluster API managed cluster.

It is recommended to manage machines with one or more MachineDeployments. MachineDeployments will transparently manage MachineSets and Machines to allow for a seamless scaling experience. A modification to the MachineDeployments spec will begin a rolling update of the machines. Follow these instructions for changing the template for an existing MachineDeployment.

MachineDeployments support different strategies for rolling out changes to Machines:

• RollingUpdate

Changes are rolled out by honouring MaxUnavailable and MaxSurge values. Only values allowed are of type Int or Strings with an integer and percentage symbol e.g “5%”.

• OnDelete

Changes are rolled out driven by the user or any entity deleting the old Machines. Only when a Machine is fully deleted a new one will come up.

For a more in-depth look at how MachineDeployments manage scaling events, take a look at the MachineDeployment controller documentation and the MachineSet controller documentation.

Support for external etcd

Cluster API Bootstrap Provider Kubeadm supports using an external etcd cluster for your workload Kubernetes clusters.

⚠️ Warnings ⚠️

Before getting started you should be aware of the expectations that come with using an external etcd cluster.

• Cluster API is unable to manage any aspect of the external etcd cluster.
• Depending on how you configure your etcd nodes you may incur additional cloud costs in data transfer.
  • As an example, cross availability zone traffic can cost money on cloud providers. You don’t have to deploy etcd across availability zones, but if you do please be aware of the costs.

Getting started

To use this, you will need to create an etcd cluster and generate an apiserver-etcd-client certificate and private key. This behaviour can be tested using kubeadm and etcdadm.

Setting up etcd with kubeadm

CA certificates are required to setup etcd cluster. If you already have a CA then the CA’s crt and key must be copied to /etc/kubernetes/pki/etcd/ca.crt and /etc/kubernetes/pki/etcd/ca.key.

If you do not already have a CA then run command kubeadm init phase certs etcd-ca. This creates two files:

• /etc/kubernetes/pki/etcd/ca.crt
• /etc/kubernetes/pki/etcd/ca.key

This certificate and private key are used to sign etcd server and peer certificates as well as other client certificates (like the apiserver-etcd-client certificate or the etcd-healthcheck-client certificate). More information on how to setup external etcd with kubeadm can be found here.

Once the etcd cluster is setup, you will need the following files from the etcd cluster:

1. /etc/kubernetes/pki/apiserver-etcd-client.crt and /etc/kubernetes/pki/apiserver-etcd-client.key
2. /etc/kubernetes/pki/etcd/ca.crt

You’ll use these files to create the necessary Secrets on the management cluster (see the “Creating the required Secrets” section).

Setting up etcd with etcdadm (Alpha)

etcdadm creates the CA if one does not exist, uses it to sign its server and peer certificates, and finally to sign the API server etcd client certificate. The CA’s crt and key generated using etcdadm are stored in /etc/etcd/pki/ca.crt and /etc/etcd/pki/ca.key. etcdadm also generates a certificate for the API server etcd client; the certificate and private key are found at /etc/etcd/pki/apiserver-etcd-client.crt and /etc/etcd/pki/apiserver-etcd-client.key, respectively.

Once the etcd cluster has been bootstrapped using etcdadm, you will need the following files from the etcd cluster:

1. /etc/etcd/pki/apiserver-etcd-client.crt and /etc/etcd/pki/apiserver-etcd-client.key
2. /etc/etcd/pki/etcd/ca.crt

You’ll use these files in the next section to create the necessary Secrets on the management cluster.

Creating the required Secrets

Regardless of the method used to bootstrap the etcd cluster, you will need to use the certificates copied from the etcd cluster to create some Kubernetes Secrets on the management cluster.

In the commands below to create the Secrets, substitute $CLUSTER_NAME with the name of the workload cluster to be created by CAPI, and substitute $CLUSTER_NAMESPACE with the name of the namespace where the workload cluster will be created. The namespace can be omitted if the workload cluster will be created in the default namespace.

First, you will need to create a Secret containing the API server etcd client certificate and key. This command assumes the certificate and private key are in the current directory; adjust your command accordingly if they are not:

# Kubernetes API server etcd client certificate and key
kubectl create secret tls $CLUSTER_NAME-apiserver-etcd-client \
  --cert apiserver-etcd-client.crt \
  --key apiserver-etcd-client.key \
  --namespace $CLUSTER_NAMESPACE

Next, create a Secret for the etcd cluster’s CA certificate. The kubectl create secret tls command requires both a certificate and a key, but the key isn’t needed by CAPI. Instead, use the kubectl create secret generic command, and note that the file containing the CA certificate must be named tls.crt:

# Etcd's CA crt file to validate the generated client certificates
kubectl create secret generic $CLUSTER_NAME-etcd \
  --from-file tls.crt \
  --namespace $CLUSTER_NAMESPACE

Note: The above commands will base64 encode the certificate/key files by default.

Alternatively you can base64 encode the files and put them in two secrets. The secrets must be formatted as follows and the cert material must be base64 encoded:

# Kubernetes APIServer etcd client certificate
kind: Secret
apiVersion: v1
metadata:
  name: $CLUSTER_NAME-apiserver-etcd-client
  namespace: $CLUSTER_NAMESPACE
data:
  tls.crt: |
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCRENDQWV5Z0F3SUJBZ0lJZFlkclZUMzV0
    NW93RFFZSktvWklodmNOQVFFTEJRQXdEekVOTUFzR0ExVUUKQXhNRVpYUmpaREFlRncweE9UQTVN
    ...
  tls.key: |
    LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb3dJQkFBS0NBUUVBdlFlTzVKOE5j
    VCtDeGRubFR3alpuQ3YwRzByY0tETklhZzlSdFdrZ1p4MEcxVm1yClA4Zy9BRkhXVHdxSTUrNi81
    ...

# Etcd's CA crt file to validate the generated client certificates
kind: Secret
apiVersion: v1
metadata:
  name: $CLUSTER_NAME-etcd
  namespace: $CLUSTER_NAMESPACE
data:
  tls.crt: |
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURBRENDQWVpZ0F3SUJBZ0lJRDNrVVczaDIy
    K013RFFZSktvWklodmNOQVFFTEJRQXdEekVOTUFzR0ExVUUKQXhNRVpYUmpaREFlRncweE9UQTVN
    ...

The Secrets must be created before creating the workload cluster.

Configuring CABPK

Once the Secrets are in place on the management cluster, the rest of the process leverages standard kubeadm configuration. Configure your ClusterConfiguration for the workload cluster as follows:

apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfig
metadata:
  name: CLUSTER_NAME-controlplane-0
  namespace: CLUSTER_NAMESPACE
spec:
  ... # initConfiguration goes here
  clusterConfiguration:
    etcd:
      external:
        endpoints:
          - https://10.0.0.230:2379
        caFile: /etc/kubernetes/pki/etcd/ca.crt
        certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
        keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
    ... # other clusterConfiguration goes here

Create your workload cluster as normal. The new workload cluster should use the configured external etcd nodes instead of creating co-located etcd Pods on the control plane nodes.

Additional Notes/Caveats

• Depending on the provider, additional changes to the workload cluster’s manifest may be necessary to ensure the new CAPI-managed nodes have connectivity to the existing etcd nodes. For example, on AWS you will need to leverage the additionalSecurityGroups field on the AWSMachine and/or AWSMachineTemplate objects to add the CAPI-managed nodes to a security group that has connectivity to the existing etcd cluster. Other mechanisms exist for other providers.

Using Kustomize with Workload Cluster Manifests

Although the clusterctl generate cluster command exposes a number of different configuration values for customizing workload cluster YAML manifests, some users may need additional flexibility above and beyond what clusterctl generate cluster or the example “flavor” templates that some CAPI providers supply (as an example, see these flavor templates for the Cluster API Provider for Azure). In the future, a templating solution may be integrated into clusterctl to help address this need, but in the meantime users can use kustomize as a solution to this need.

This document provides a few examples of using kustomize with Cluster API. All of these examples assume that you are using a directory structure that looks something like this:

.
├── base
│   ├── base.yaml
│   └── kustomization.yaml
└── overlays
    ├── custom-ami
    │   ├── custom-ami.json
    │   └── kustomization.yaml
    └── mhc
        ├── kustomization.yaml
        └── workload-mhc.yaml

In the overlay directories, the “base” (unmodified) Cluster API configuration (perhaps generated using clusterctl generate cluster) would be referenced as a resource in kustomization.yaml using ../../base.

Example: Using Kustomize to Specify Custom Images

Users can use kustomize to specify custom OS images for Cluster API nodes. Using the Cluster API Provider for AWS (CAPA) as an example, the following kustomization.yaml would leverage a JSON 6902 patch to modify the AMI for nodes in a workload cluster:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patchesJson6902:
  - path: custom-ami.json
    target:
      group: infrastructure.cluster.x-k8s.io
      kind: AWSMachineTemplate
      name: ".*"
      version: v1alpha3

The referenced JSON 6902 patch in custom-ami.json would look something like this:

[
    { "op": "add", "path": "/spec/template/spec/ami", "value": "ami-042db61632f72f145"}
]

This configuration assumes that the workload cluster only uses MachineDeployments. Since MachineDeployments and the KubeadmControlPlane both leverage AWSMachineTemplates, this kustomize configuration would catch all nodes in the workload cluster.

Example: Adding a MachineHealthCheck for a Workload Cluster

Users could also use kustomize to combine additional resources, like a MachineHealthCheck (MHC), with the base Cluster API manifest. In an overlay directory, specify the following in kustomization.yaml:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
  - workload-mhc.yaml

The content of the workload-mhc.yaml file would be the definition of a standard MHC:

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineHealthCheck
metadata:
  name: md-0-mhc
spec:
  clusterName: test
  # maxUnhealthy: 40%
  nodeStartupTimeout: 10m
  selector:
    matchLabels:
      cluster.x-k8s.io/deployment-name: md-0
  unhealthyNodeConditions:
  - type: Ready
    status: Unknown
    timeout: 300s
  - type: Ready
    status: "False"
    timeout: 300s
  unhealthyMachineConditions:
  - type: "NodeReady"
    status: Unknown
    timeout: 1800s
  - type: "InfrastructureReady"
    status: "False"
    timeout: 1800s

You would want to ensure the clusterName field in the MachineHealthCheck manifest appropriately matches the name of the workload cluster, taking into account any transformations you may have specified in kustomization.yaml (like the use of “namePrefix” or “nameSuffix”).

Running kustomize build . with this configuration would append the MHC to the base Cluster API manifest, thus creating the MHC at the same time as the workload cluster.

Modifying Names

The kustomize “namePrefix” and “nameSuffix” transformers are not currently “Cluster API aware.” Although it is possible to use these transformers with Cluster API manifests, doing so requires separate patches for Clusters versus infrastructure-specific equivalents (like an AzureCluster or a vSphereCluster). This can significantly increase the complexity of using kustomize for this use case.

Modifying the transformer configurations for kustomize can make it more effective with Cluster API. For example, changes to the nameReference transformer in kustomize will enable kustomize to know about the references between Cluster API objects in a manifest. See here for more information on transformer configurations.

Add the following content to the namereference.yaml transformer configuration:

- kind: Cluster
  group: cluster.x-k8s.io
  version: v1beta2
  fieldSpecs:
  - path: spec/clusterName
    kind: MachineDeployment
  - path: spec/template/spec/clusterName
    kind: MachineDeployment

- kind: AWSCluster
  group: infrastructure.cluster.x-k8s.io
  version: v1alpha3
  fieldSpecs:
  - path: spec/infrastructureRef/name
    kind: Cluster

- kind: KubeadmControlPlane
  group: controlplane.cluster.x-k8s.io
  version: v1alpha3
  fieldSpecs:
  - path: spec/controlPlaneRef/name
    kind: Cluster

- kind: AWSMachine
  group: infrastructure.cluster.x-k8s.io
  version: v1alpha3
  fieldSpecs:
  - path: spec/infrastructureRef/name
    kind: Machine

- kind: KubeadmConfig
  group: bootstrap.cluster.x-k8s.io
  version: v1alpha3
  fieldSpecs:
  - path: spec/bootstrap/configRef/name
    kind: Machine

- kind: AWSMachineTemplate
  group: infrastructure.cluster.x-k8s.io
  version: v1alpha3
  fieldSpecs:
  - path: spec/template/spec/infrastructureRef/name
    kind: MachineDeployment
  - path: spec/infrastructureTemplate/name
    kind: KubeadmControlPlane

- kind: KubeadmConfigTemplate
  group: bootstrap.cluster.x-k8s.io
  version: v1alpha3
  fieldSpecs:
  - path: spec/template/spec/bootstrap/configRef/name
    kind: MachineDeployment

Including this custom configuration in a kustomization.yaml would then enable the use of simple “namePrefix” and/or “nameSuffix” directives, like this:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
configurations:
  - namereference.yaml
namePrefix: "blue-"
nameSuffix: "-dev"

Running kustomize build . with this configuration would modify the name of all the Cluster API objects and the associated referenced objects, adding “blue-” at the beginning and appending “-dev” at the end.

Upgrading Cluster API components

When to upgrade

In general, it’s recommended to upgrade to the latest version of Cluster API to take advantage of bug fixes, new features and improvements.

Considerations

If moving between different API versions, there may be additional tasks that you need to complete. See below for detailed instructions.

Ensure that the version of Cluster API is compatible with the Kubernetes version of the management cluster.

Upgrading to newer versions of 1.0.x

Use clusterctl to upgrade between versions of Cluster API 1.0.x.

Control Plane Management

This section provides details about control plane providers.

Kubeadm control plane

Using the Kubeadm control plane type to manage a control plane provides several ways to upgrade control plane machines.

Kubeconfig management

KCP will generate and manage the admin Kubeconfig for clusters. The client certificate for the admin user is created with a valid lifespan of a year, and will be automatically regenerated when the cluster is reconciled and has less than 6 months of validity remaining.

Upgrades

See the section on upgrading clusters.

Running workloads on control plane machines

We don’t suggest running workloads on control planes, and highly encourage avoiding it unless absolutely necessary.

However, in the case the user wants to run non-control plane workloads on control plane machines they are ultimately responsible for ensuring the proper functioning of those workloads, given that KCP is not aware of the specific requirements for each type of workload (e.g. preserving quorum, shutdown procedures etc.).

In order to do so, the user could leverage on the same assumption that applies to all the Cluster API Machines:

• The Kubernetes node hosted on the Machine will be cordoned & drained before removal (with well known exceptions like full Cluster deletion).
• The Machine will respect PreDrainDeleteHook and PreTerminateDeleteHook. see the Machine Deletion Phase Hooks proposal for additional details.

In-place propagation

Changes to the following fields of KubeadmControlPlane are propagated in-place to the Machines and do not trigger a full rollout:

• .spec.machineTemplate.metadata.labels
• .spec.machineTemplate.metadata.annotations
• .spec.nodeDrainTimeout
• .spec.nodeDeletionTimeout
• .spec.nodeVolumeDetachTimeout

Changes to the following fields of KubeadmControlPlane are propagated in-place to the InfrastructureMachine and KubeadmConfig:

• .spec.machineTemplate.metadata.labels
• .spec.machineTemplate.metadata.annotations

Note: Changes to these fields will not be propagated to Machines, InfraMachines and KubeadmConfigs that are marked for deletion (example: because of scale down).

MicroK8s control plane provider

What is the Cluster API MicroK8s control plane provider ?

Cluster API MicroK8s control plane provider (CACPM) is a component responsible for managing the control plane of the provisioned clusters. This implementation uses MicroK8s for cluster provisioning and management.

Currently the CACPM does not expose any functionality. It serves however the following purposes:

• Sets the ProviderID on the provisioned nodes. MicroK8s will not set the provider ID automatically so the control plane provider identifies the VMs’ provider IDs and updates the respective machine objects.
• Updates the machine state.
• Generates and provisions the kubeconfig file used for accessing the cluster. The kubeconfig file is stored as a secret and the user can retrieve via clusterctl.

Updating Machine Infrastructure and Bootstrap Templates

Updating Infrastructure Machine Templates

Several different components of Cluster API leverage infrastructure machine templates, including KubeadmControlPlane, MachineDeployment, and MachineSet. These MachineTemplate resources should be immutable, unless the infrastructure provider documentation indicates otherwise for certain fields (see below for more details).

The correct process for modifying an infrastructure machine template is as follows:

1. Duplicate an existing template. Users can use kubectl get <MachineTemplateType> <name> -o yaml > file.yaml to retrieve a template configuration from a running cluster to serve as a starting point.
2. Update the desired fields. Fields that might need to be modified could include the SSH key, the AWS instance type, or the Azure VM size. Refer to the provider-specific documentation for more details on the specific fields that each provider requires or accepts.
3. Give the newly-modified template a new name by modifying the metadata.name field (or by using metadata.generateName).
4. Create the new infrastructure machine template on the API server using kubectl. (If the template was initially created using the command in step 1, be sure to clear out any extraneous metadata, including the resourceVersion field, before trying to send it to the API server.)

Once the new infrastructure machine template has been persisted, users may modify the object that was referencing the infrastructure machine template. For example, to modify the infrastructure machine template for the KubeadmControlPlane object, users would modify the spec.infrastructureTemplate.name field. For a MachineDeployment, users would need to modify the spec.template.spec.infrastructureRef.name field and the controller would orchestrate the upgrade by managing MachineSets pointing to the new and old references. In the case of a MachineSet with no MachineDeployment owner, if its template reference is changed, it will only affect upcoming Machines.

In all cases, the name field should be updated to point to the newly-modified infrastructure machine template. This will trigger a rolling update. (This same process is described in the documentation for upgrading the underlying machine image for KubeadmControlPlane in the “How to upgrade the underlying machine image” section.)

Some infrastructure providers may, at their discretion, choose to support in-place modifications of certain infrastructure machine template fields. This may be useful if an infrastructure provider is able to make changes to running instances/machines, such as updating allocated memory or CPU capacity. In such cases, however, Cluster API will not trigger a rolling update.

Updating Bootstrap Templates

Several different components of Cluster API leverage bootstrap templates, including MachineDeployment, and MachineSet. When used in MachineDeployment or MachineSet changes to those templates do not trigger rollouts of already existing Machines. New Machines are created based on the current version of the bootstrap template.

The correct process for modifying a bootstrap template is as follows:

1. Duplicate an existing template. Users can use kubectl get <BootstrapTemplateType> <name> -o yaml > file.yaml to retrieve a template configuration from a running cluster to serve as a starting point.
2. Update the desired fields.
3. Give the newly-modified template a new name by modifying the metadata.name field (or by using metadata.generateName).
4. Create the new bootstrap template on the API server using kubectl. (If the template was initially created using the command in step 1, be sure to clear out any extraneous metadata, including the resourceVersion field, before trying to send it to the API server.)

Once the new bootstrap template has been persisted, users may modify the object that was referencing the bootstrap template. For example, to modify the bootstrap template for the MachineDeployment object, users would modify the spec.template.spec.bootstrap.configRef.name field. The name field should be updated to point to the newly-modified bootstrap template. This will trigger a rolling update.

Workload bootstrap using GitOps

Cluster API can be utilized in combination with the Cluster API addon provider for helm (CAAPH) to install and configure a GitOps agent and then the GitOps agent hydrates clusters automatically with various workloads.

Prerequisites

Follow the quickstart setup guide for your provider but ensure that CAAPH is installed via including the addon=helm with either:

1. clusterctl using clusterctl init --infrastructure ### --addon helm or
2. Cluster API Operator using helm install capi-operator capi-operator/cluster-api-operator ... --set infrastructure=#### --set addon=helm

Bootstrap ManagedCluster using ArgoCD

Add the labels argoCDChart: enabled and guestbook: enabled to your desired workload cluster yaml file in the Cluster metadata section, for example:

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: my-cluster
  namespace: default
  labels:
    argoCDChart: enabled
    guestbook: enabled

Then create and kubectl apply -f the following file on the management cluster to install the ArgoCD agent and the sample guestbook app to the workload cluster via the argo helm charts using CAAPH:

apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: argocd
spec:
  clusterSelector:
    matchLabels:
      argoCDChart: enabled
  repoURL: https://argoproj.github.io/argo-helm
  chartName: argo-cd
  options:
    waitForJobs: true
    wait: true
    timeout: 5m
    install:
      createNamespace: true
---
apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: argocdguestbook
spec:
  clusterSelector:
    matchLabels:
      guestbook: enabled
  repoURL: https://argoproj.github.io/argo-helm
  chartName: argocd-apps
  options:
    waitForJobs: true
    wait: true
    timeout: 5m
    install:
      createNamespace: true
  valuesTemplate: |
    applications:
      - name: guestbook
        namespace: argocd
        finalizers:
        - resources-finalizer.argocd.argoproj.io
        project: default
        sources:
          - repoURL: https://github.com/argoproj/argocd-example-apps.git
            path: guestbook
            targetRevision: HEAD
        destination:
          server: https://kubernetes.default.svc
          namespace: guestbook
        syncPolicy:
          automated:
            prune: false
            selfHeal: false
          syncOptions:
          - CreateNamespace=true
        revisionHistoryLimit: null
    ignoreDifferences:
      - group: apps
        kind: Deployment
        jsonPointers:
        - /spec/replicas
    info:
    - name: url
      value: https://argoproj.github.io/

This will automatically install ArgoCD in the ArgoCD namespace and the guestbook application into the guestbook namespace. Adding or labeling additional clusters with argoCDChart: enabled and guestbook: enabled will automatically install the ArgoCD agent and the guestbook application and there is no need to create additional CAAPH HelmChartProxy entries.

The ArgoCD console can be viewed by connecting to the workload cluster and then doing the following:

# Get the admin password
kubectl get secrets argocd-initial-admin-secret -n argocd --template="{{index .data.password | base64decode}}"
kubectl port-forward service/capiargo-argocd-server -n default 8080:443
# and then open the browser on http://localhost:8080 and accept the certificate

The Guestbook application deployment can be seen once logged into the ArgoCD console. Since the GitOps agent points to the git repository, any changes to the repository will automatically update the workload cluster. The git repository could be configured to utilize the App of Apps pattern to install all platform requirements for the cluster. The App of Apps pattern is a single application that installs all other applications and configurations for the cluster.

This same pattern could also utilize the Flux agent using the Flux helm charts being installed and configured by CAAPH.

Automated Machine management

This section details some tasks related to automated Machine management.

• Scaling
• Autoscaling
• Healthchecking
• Machine deletion process

Scaling Nodes

This section applies only to worker Machines. You can add or remove compute capacity for your cluster workloads by creating or removing Machines. A Machine expresses intent to have a Node with a defined form factor.

Machines can be owned by scalable resources i.e. MachineSet and MachineDeployments.

You can scale MachineSets and MachineDeployments in or out by expressing intent via .spec.replicas or updating the scale subresource e.g kubectl scale machinedeployment foo --replicas=5.

If you need to prioritize which Machines get deleted during scale-down, add the cluster.x-k8s.io/delete-machine label to the Machine. KCP or a MachineSet will delete labeled control plane or worker Machines first, and this label has top priority over all delete policies.

Note: The label only affects MachineSet scale-down; in a MachineDeployment, the choice of MachineSet to scale-down may bypass labeled Machines.

When you delete a Machine directly or by scaling down, the same process takes place in the same order:

• The Node backed by that Machine will try to be drained indefinitely and will wait for any volume to be detached from the Node unless you specify a .spec.nodeDrainTimeout.
  • CAPI uses default kubectl draining implementation with -–ignore-daemonsets=true. If you needed to ensure DaemonSets eviction you’d need to do so manually by also adding proper taints to avoid rescheduling.
• The infrastructure backing that Node will try to be deleted indefinitely.
• Only when the infrastructure is gone, the Node will try to be deleted indefinitely unless you specify .spec.nodeDeletionTimeout.

Using the Cluster Autoscaler

This section applies only to worker Machines. Cluster Autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster based on the utilization of Pods and Nodes in your cluster. For more general information about the Cluster Autoscaler, please see the project documentation.

The following instructions are a reproduction of the Cluster API provider specific documentation from the Autoscaler project documentation.

Cluster Autoscaler on Cluster API

The cluster autoscaler on Cluster API uses the cluster-api project to manage the provisioning and de-provisioning of nodes within a Kubernetes cluster.

Table of Contents:

• Kubernetes Version
• Starting the Autoscaler
• Configuring node group auto discovery
• Connecting cluster-autoscaler to Cluster API management and workload Clusters
  • Autoscaler running in a joined cluster using service account credentials
  • Autoscaler running in workload cluster using service account credentials, with separate management cluster
  • Autoscaler running in management cluster using service account credentials, with separate workload cluster
  • Autoscaler running anywhere, with separate kubeconfigs for management and workload clusters
  • Autoscaler running anywhere, with a common kubeconfig for management and workload clusters
• Enabling Autoscaling
  • Scale from zero support
    • RBAC changes for scaling from zero
    • Pre-defined labels and taints on nodes scaled from zero
    • CPU Architecture awareness for single-arch clusters
• Specifying a Custom Resource Group
• Specifying a Custom Resource Version
• Sample manifest
  • A note on permissions
• Autoscaling with ClusterClass and Managed Topologies
• Special note on GPU instances
• Special note on balancing similar node groups

Kubernetes Version

The cluster-api provider requires Kubernetes v1.16 or greater to run the v1alpha3 version of the API.

Starting the Autoscaler

To enable the Cluster API provider, you must first specify it in the command line arguments to the cluster autoscaler binary. For example:

cluster-autoscaler --cloud-provider=clusterapi

Please note, this example only shows the cloud provider options, you will most likely need other command line flags. For more information you can invoke cluster-autoscaler --help to see a full list of options.

Configuring node group auto discovery

You must configure node group auto discovery to inform cluster autoscaler which cluster in which to find for scalable node groups.

Limiting cluster autoscaler to only match against resources in the blue namespace

--node-group-auto-discovery=clusterapi:namespace=blue

Limiting cluster autoscaler to only match against resources belonging to Cluster test1

--node-group-auto-discovery=clusterapi:clusterName=test1

Limiting cluster autoscaler to only match against resources matching the provided labels

--node-group-auto-discovery=clusterapi:color=green,shape=square

These can be mixed and matched in any combination, for example to only match resources in the staging namespace, belonging to the purple cluster, with the label owner=jim:

--node-group-auto-discovery=clusterapi:namespace=staging,clusterName=purple,owner=jim

Connecting cluster-autoscaler to Cluster API management and workload Clusters

Important

--cloud-config is the flag for specifying a mount volume path to the kubernetes configuration (ie KUBECONFIG) to the cluster-autoscaler for communicating with the cluster-api management cluster for the purpose of scaling machines.

Important

``–kubeconfig` is the flag for specifying a mount volume path to the kubernetes configuration (ie KUBECONFIG) to the cluster-autoscaler for communicating with the cluster-api workload cluster for the purpose of watching Nodes and Pods. This flag can be affected by the desired topology for deploying the cluster-autoscaler, please see the diagrams below for more information.

You will also need to provide the path to the kubeconfig(s) for the management and workload cluster you wish cluster-autoscaler to run against. To specify the kubeconfig path for the workload cluster to monitor, use the --kubeconfig option and supply the path to the kubeconfig. If the --kubeconfig option is not specified, cluster-autoscaler will attempt to use an in-cluster configuration. To specify the kubeconfig path for the management cluster to monitor, use the --cloud-config option and supply the path to the kubeconfig. If the --cloud-config option is not specified it will fall back to using the kubeconfig that was provided with the --kubeconfig option.

Autoscaler running in a joined cluster using service account credentials

+-----------------+
| mgmt / workload |
| --------------- |
|    autoscaler   |
+-----------------+

Use in-cluster config for both management and workload cluster:

cluster-autoscaler --cloud-provider=clusterapi

Autoscaler running in workload cluster using service account credentials, with separate management cluster

+--------+              +------------+
|  mgmt  |              |  workload  |
|        | cloud-config | ---------- |
|        |<-------------+ autoscaler |
+--------+              +------------+

Use in-cluster config for workload cluster, specify kubeconfig for management cluster:

cluster-autoscaler --cloud-provider=clusterapi \
                   --cloud-config=/mnt/kubeconfig

Autoscaler running in management cluster using service account credentials, with separate workload cluster

+------------+             +----------+
|    mgmt    |             | workload |
| ---------- | kubeconfig  |          |
| autoscaler +------------>|          |
+------------+             +----------+

Use in-cluster config for management cluster, specify kubeconfig for workload cluster:

cluster-autoscaler --cloud-provider=clusterapi \
                   --kubeconfig=/mnt/kubeconfig \
                   --clusterapi-cloud-config-authoritative

Autoscaler running anywhere, with separate kubeconfigs for management and workload clusters

+--------+               +------------+             +----------+
|  mgmt  |               |     ?      |             | workload |
|        |  cloud-config | ---------- | kubeconfig  |          |
|        |<--------------+ autoscaler +------------>|          |
+--------+               +------------+             +----------+

Use separate kubeconfigs for both management and workload cluster:

cluster-autoscaler --cloud-provider=clusterapi \
                   --kubeconfig=/mnt/workload.kubeconfig \
                   --cloud-config=/mnt/management.kubeconfig

Autoscaler running anywhere, with a common kubeconfig for management and workload clusters

+---------------+             +------------+
| mgmt/workload |             |     ?      |
|               |  kubeconfig | ---------- |
|               |<------------+ autoscaler |
+---------------+             +------------+

Use a single provided kubeconfig for both management and workload cluster:

cluster-autoscaler --cloud-provider=clusterapi \
                   --kubeconfig=/mnt/workload.kubeconfig

Enabling Autoscaling

To enable the automatic scaling of components in your cluster-api managed cloud there are a few annotations you need to provide. These annotations must be applied to either MachineSet, MachineDeployment, or MachinePool resources depending on the type of cluster-api mechanism that you are using.

There are two annotations that control how a cluster resource should be scaled:

• cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size - This specifies the minimum number of nodes for the associated resource group. The autoscaler will not scale the group below this number. Please note that the cluster-api provider will not scale down to, or from, zero unless that capability is enabled (see Scale from zero support).

• cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size - This specifies the maximum number of nodes for the associated resource group. The autoscaler will not scale the group above this number.

The autoscaler will monitor any MachineSet, MachineDeployment, or MachinePool containing both of these annotations.

Note: The cluster autoscaler does not enforce the node group sizes. If a node group is below the minimum number of nodes, or above the maximum number of nodes, the cluster autoscaler will not scale that node group up or down. The cluster autoscaler can be configured to enforce the minimum node group size by enabling the --enforce-node-group-min-size flag. Please see this entry in the Cluster Autoscaler FAQ for more information.

Note: MachinePool support in cluster-autoscaler requires a provider implementation that supports the “MachinePool Machines” feature.

Scale from zero support

The Cluster API community has defined an opt-in method for infrastructure providers to enable scaling from zero-sized node groups in the Opt-in Autoscaling from Zero enhancement. As defined in the enhancement, each provider may add support for scaling from zero to their provider, but they are not required to do so. If you are expecting built-in support for scaling from zero, please check with the Cluster API infrastructure providers that you are using.

If your Cluster API provider does not have support for scaling from zero, you may still use this feature through the capacity annotations. You may add these annotations to your MachineDeployments, or MachineSets if you are not using MachineDeployments (it is not needed on both), to instruct the cluster autoscaler about the sizing of the nodes in the node group. At the minimum, you must specify the CPU and memory annotations, these annotations should match the expected capacity of the nodes created from the infrastructure.

Note: The scale from zero annotations will override any capacity information supplied by the Cluster API provider in the infrastructure machine templates. If both the annotations and the provider supplied capacity information are present, the annotations will take precedence.

For example, if my MachineDeployment will create nodes that have “16000m” CPU, “128G” memory, “100Gi” ephemeral disk storage, 2 NVidia GPUs, and can support 200 max pods, the following annotations will instruct the autoscaler how to expand the node group from zero replicas:

apiVersion: cluster.x-k8s.io/v1alpha4
kind: MachineDeployment
metadata:
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    capacity.cluster-autoscaler.kubernetes.io/memory: "128G"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "16"
    capacity.cluster-autoscaler.kubernetes.io/ephemeral-disk: "100Gi"
    capacity.cluster-autoscaler.kubernetes.io/maxPods: "200"
    // Device Plugin
    // Comment out the below annotation if DRA is enabled on your cluster running k8s v1.32.0 or greater
    capacity.cluster-autoscaler.kubernetes.io/gpu-type: "nvidia.com/gpu"
    // Dynamic Resource Allocation (DRA)
    // Uncomment the below annotation if DRA is enabled on your cluster running k8s v1.32.0 or greater
    // capacity.cluster-autoscaler.kubernetes.io/dra-driver: "gpu.nvidia.com"
    // Common in Device Plugin and DRA
    capacity.cluster-autoscaler.kubernetes.io/gpu-count: "2"

Note: the maxPods annotation will default to 110 if it is not supplied. This value is inspired by the Kubernetes best practices Considerations for large clusters.

Note: User should select the annotation for GPU either gpu-type or dra-driver depends on whether using Device Plugin or Dynamic Resource Allocation(DRA). gpu-count is a common parameter in both.

RBAC changes for scaling from zero

If you are using the opt-in support for scaling from zero as defined by the Cluster API infrastructure provider, you will need to add the infrastructure machine template types to your role permissions for the service account associated with the cluster autoscaler deployment. The service account will need permission to get, list, and watch the infrastructure machine templates for your infrastructure provider.

For example, when using the Kubemark provider you will need to set the following permissions:

rules:
  - apiGroups:
    - infrastructure.cluster.x-k8s.io
    resources:
    - kubemarkmachinetemplates
    verbs:
    - get
    - list
    - watch

Pre-defined labels and taints on nodes scaled from zero

Taints for scale from zero can be configured in two ways, listed below in order of precedence (highest first):

1. Capacity annotation (highest priority)

The capacity.cluster-autoscaler.kubernetes.io/taints annotation accepts a comma-separated list of taints and always takes precedence over taints defined in the scalable resource spec.

apiVersion: cluster.x-k8s.io/v1alpha4
kind: MachineDeployment
metadata:
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    capacity.cluster-autoscaler.kubernetes.io/memory: "128G"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "16"
    capacity.cluster-autoscaler.kubernetes.io/labels: "key1=value1,key2=value2"
    capacity.cluster-autoscaler.kubernetes.io/taints: "key1=value1:NoSchedule,key2=value2:NoExecute"

2. Scalable resource spec (requires CAPI v1.12+ with MachineTaintPropagation feature gate enabled)

When the MachineTaintPropagation feature gate is enabled in Cluster API, taints defined in spec.template.spec.taints of a MachineSet, MachineDeployment, or MachinePool are read directly by the cluster autoscaler. If an annotation taint has the same key and effect as a spec taint, the annotation value takes precedence.

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    capacity.cluster-autoscaler.kubernetes.io/memory: "128G"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "16"
    # Override the value of the "dedicated" taint defined in spec below.
    capacity.cluster-autoscaler.kubernetes.io/taints: "dedicated=gpu-override:NoSchedule"
spec:
  template:
    spec:
      taints:
        - key: dedicated
          value: gpu
          effect: NoSchedule
          propagation: Always
        - key: node-setup
          value: "true"
          effect: NoSchedule
          propagation: Always

Note: For labels, the capacity annotation values are merged with the labels propagated from the scalable Cluster API resource. If the same label key is defined in both, the annotation value takes precedence. Please see the Cluster API Book chapter on Metadata propagation for more information.

For taints, annotation taints are merged with spec taints. If the same key and effect is defined in both, the annotation value takes precedence. Spec taints without a matching annotation taint are preserved.

Pre-defined csi driver information on nodes scaled from zero

To provide CSI driver information for scale from zero, the optional capacity annotation may be supplied as a comma separated list of driver name and volume limit key/value pairs, as demonstrated in the example below:

apiVersion: cluster.x-k8s.io/v1alpha4
kind: MachineDeployment
metadata:
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "5"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "0"
    capacity.cluster-autoscaler.kubernetes.io/memory: "128G"
    capacity.cluster-autoscaler.kubernetes.io/cpu: "16"
    capacity.cluster-autoscaler.kubernetes.io/csi-driver: "ebs.csi.aws.com=25,efs.csi.aws.com=16"

Note: The CSI driver information supplied through the capacity annotation specifies which CSI drivers will be installed on nodes scaled from zero, along with their respective volume limits. The format is driver-name=volume-limit with multiple drivers separated by commas.

Per-NodeGroup autoscaling options

Custom autoscaling options per node group (MachineDeployment/MachinePool/MachineSet) can be specified as annoations with a common prefix:

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachineDeployment
metadata:
  annotations:
    # overrides --scale-down-utilization-threshold global value for that specific MachineDeployment
    cluster.x-k8s.io/autoscaling-options-scaledownutilizationthreshold: "0.5"
    # overrides --scale-down-gpu-utilization-threshold global value for that specific MachineDeployment
    cluster.x-k8s.io/autoscaling-options-scaledowngpuutilizationthreshold: "0.5"
    # overrides --scale-down-unneeded-time global value for that specific MachineDeployment
    cluster.x-k8s.io/autoscaling-options-scaledownunneededtime: "10m0s"
    # overrides --scale-down-unready-time global value for that specific MachineDeployment
    cluster.x-k8s.io/autoscaling-options-scaledownunreadytime: "20m0s"
    # overrides --max-node-provision-time global value for that specific MachineDeployment
    cluster.x-k8s.io/autoscaling-options-maxnodeprovisiontime: "20m0s"
    # overrides --max-node-startup-time global value for that specific MachineDeployment
    cluster.x-k8s.io/autoscaling-options-maxnodestartuptime: "20m0s"

CPU Architecture awareness for single-arch clusters

Users of single-arch non-amd64 clusters who are using scale from zero support should also set the CAPI_SCALE_ZERO_DEFAULT_ARCH environment variable to set the architecture of the nodes they want to default the node group templates to. The autoscaler will default to amd64 if it is not set, and the node group templates may not match the nodes’ architecture, specifically when the workload triggering the scale-up uses a node affinity predicate checking for the node’s architecture.

Specifying a Custom Resource Group

By default all Kubernetes resources consumed by the Cluster API provider will use the group cluster.x-k8s.io, with a dynamically acquired version. In some situations, such as testing or prototyping, you may wish to change this group variable. For these situations you may use the environment variable CAPI_GROUP to change the group that the provider will use.

Please note that setting the CAPI_GROUP environment variable will also cause the annotations for minimum and maximum size to change. This behavior will also affect the machine annotation on nodes, the machine deletion annotation, and the cluster name label. For example, if CAPI_GROUP=test.k8s.io then the minimum size annotation key will be test.k8s.io/cluster-api-autoscaler-node-group-min-size, the machine annotation on nodes will be test.k8s.io/machine, the machine deletion annotation will be test.k8s.io/delete-machine, and the cluster name label will be test.k8s.io/cluster-name.

Specifying a Custom Resource Version

When determining the group version for the Cluster API types, by default the autoscaler will look for the latest version of the group. For example, if MachineDeployments exist in the cluster.x-k8s.io group at versions v1alpha1 and v1beta1, the autoscaler will choose v1beta1.

In some cases it may be desirable to specify which version of the API the cluster autoscaler should use. This can be useful in debugging scenarios, or in situations where you have deployed multiple API versions and wish to ensure that the autoscaler uses a specific version.

Setting the CAPI_VERSION environment variable will instruct the autoscaler to use the version specified. This works in a similar fashion as the API group environment variable with the exception that there is no default value. When this variable is not set, the autoscaler will use the behavior described above.

Sample manifest

A sample manifest that will create a deployment running the autoscaler is available. It can be deployed by passing it through envsubst, providing these environment variables to set the namespace to deploy into as well as the image and tag to use:

export AUTOSCALER_NS=kube-system
export AUTOSCALER_IMAGE=registry.k8s.io/autoscaling/cluster-autoscaler:v1.29.0
envsubst < examples/deployment.yaml | kubectl apply -f-

A note on permissions

The cluster-autoscaler-management role for accessing cluster api scalable resources is scoped to ClusterRole. This may not be ideal for all environments (eg. Multi tenant environments). In such cases, it is recommended to scope it to a Role mapped to a specific namespace.

Autoscaling with ClusterClass and Managed Topologies

For users using ClusterClass and Managed Topologies the Cluster Topology controller attempts to set MachineDeployment replicas based on the spec.topology.workers.machineDeployments[].replicas field. In order to use the Cluster Autoscaler this field can be left unset in the Cluster definition.

The below Cluster definition shows which field to leave unset:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: "my-cluster"
  namespace: default
spec:
  clusterNetwork:
    services:
      cidrBlocks: ["10.128.0.0/12"]
    pods:
      cidrBlocks: ["192.168.0.0/16"]
    serviceDomain: "cluster.local"
  topology:
    class: "quick-start"
    version: v1.24.0
    controlPlane:
      replicas: 1
    workers:
      machineDeployments:
        - class: default-worker
          name: linux
       ## replicas field is not set.
       ## replicas: 1

If the replica field is unset in the Cluster definition Autoscaling can be enabled as described above

Special note on GPU instances

As with other providers, if the device plugin on nodes that provides GPU resources takes some time to advertise the GPU resource to the cluster, this may cause Cluster Autoscaler to unnecessarily scale out multiple times.

To avoid this, you can configure kubelet on your GPU nodes to label the node before it joins the cluster by passing it the --node-labels flag. For the CAPI cloudprovider, the label format is as follows:

cluster-api/accelerator=<gpu-type>

<gpu-type> is arbitrary.

It is important to note that if you are using the --gpu-total flag to limit the number of GPU resources in your cluster that the <gpu-type> value must match between the command line flag and the node labels. Setting these values incorrectly can lead to the autoscaler creating too many GPU resources.

For example, if you are using the autoscaler command line flag --gpu-total=gfx-hardware:1:2 to limit the number of gfx-hardware resources to a minimum of 1 and maximum of 2, then you should use the kubelet node label flag --node-labels=cluster-api/accelerator=gfx-hardware.

Special note on balancing similar node groups

The Cluster Autoscaler feature to enable balancing similar node groups (activated with the --balance-similar-node-groups flag) is a powerful and popular feature. When enabled, the Cluster Autoscaler will attempt to create new nodes by adding them in a manner that balances the creation between similar node groups. With Cluster API, these node groups correspond directly to the scalable resources associated (usually MachineDeployments and MachineSets) with the nodes in question. In order for the nodes of these scalable resources to be considered similar by the Cluster Autoscaler, they must have the same capacity, labels, and taints for the nodes which will be created from them.

To help assist the Cluster Autoscaler in determining which node groups are similar, the command line flags --balancing-ignore-label and --balancing-label are provided. For an expanded discussion about balancing similar node groups and the options which are available, please see the Cluster Autoscaler FAQ.

Because Cluster API can address many different cloud providers, it is important to configure the balancing labels to ignore provider-specific labels which are used for carrying zonal information on Kubernetes nodes. The Cluster Autoscaler implementation for Cluster API does not assume any labels (aside from the well-known Kubernetes labels) to be ignored when running. Users must configure their Cluster Autoscaler deployment to ignore labels which might be different between nodes, but which do not otherwise affect node behavior or size (for example when two MachineDeployments are the same except for their deployment zones). The Cluster API community has decided not to carry cloud provider specific labels in the Cluster Autoscaler to reduce the possibility for labels to clash between providers. Additionally, the community has agreed to promote documentation and the use of the --balancing-ignore-label flag as the preferred method of deployment to reduce the extended need for maintenance on the Cluster Autoscaler when new providers are added or updated. For further context around this decision, please see the Cluster API Deep Dive into Cluster Autoscaler Node Group Balancing discussion from 2022-09-12.

The following table shows some of the most common labels used by cloud providers to designate regional or zonal information on Kubernetes nodes. It is shared here as a reference for users who might be deploying on these infrastructures.

Configure a MachineHealthCheck

Prerequisites

Before attempting to configure a MachineHealthCheck, you should have a working management cluster with at least one MachineDeployment or MachineSet deployed.

What is a MachineHealthCheck?

A MachineHealthCheck is a resource within the Cluster API which allows users to define conditions under which Machines within a Cluster should be considered unhealthy. A MachineHealthCheck is defined on a management cluster and scoped to a particular workload cluster.

When defining a MachineHealthCheck, users specify a timeout for each of the conditions that they define to check on the Machine’s Node. If any of these conditions are met for the duration of the timeout, the Machine will be remediated. By default, the action of remediating a Machine should trigger a new Machine to be created to replace the failed one, but providers are allowed to plug in more sophisticated external remediation solutions.

Creating a MachineHealthCheck

Use the following example as a basis for creating a MachineHealthCheck for worker nodes:

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineHealthCheck
metadata:
  name: capi-quickstart-node-unhealthy-5m
spec:
  # clusterName is required to associate this MachineHealthCheck with a particular cluster
  clusterName: capi-quickstart
  # selector is used to determine which Machines should be health checked
  selector:
    matchLabels:
      nodepool: nodepool-0
  # checks are the checks that are used to evaluate if a Machine is healthy.
  checks:
      # (Optional) nodeStartupTimeout determines how long a MachineHealthCheck should wait for
      # a Node to join the cluster, before considering a Machine unhealthy.
      # Defaults to 10 minutes if not specified.
      # Set to 0 to disable the node startup timeout.
      # Disabling this timeout will prevent a Machine from being considered unhealthy when
      # the Node it created has not yet registered with the cluster. This can be useful when
      # Nodes take a long time to start up or when you only want condition based checks for
      # Machine health.
      nodeStartupTimeoutSeconds: 600
    
      # Conditions to check on Nodes for matched Machines, if any condition is matched for the duration of its timeout, the Machine is considered unhealthy
      unhealthyNodeConditions:
      - type: Ready
        status: Unknown
        timeoutSeconds: 300
      - type: Ready
        status: "False"
        timeoutSeconds: 300
      unhealthyMachineConditions:
      - type: "NodeReady"
        status: Unknown
        timeoutSeconds: 1800
      - type: "InfrastructureReady"
        status: "False"
        timeoutSeconds: 1800
  # remediation configures if and how remediation is triggered if a Machine is unhealthy.
  remediation:
    triggerIf:
      # (Optional) unhealthyLessThanOrEqualTo prevents further remediation if the cluster is already partially unhealthy
      unhealthyLessThanOrEqualTo: 40%

Use this example as the basis for defining a MachineHealthCheck for control plane nodes managed via the KubeadmControlPlane:

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineHealthCheck
metadata:
  name: capi-quickstart-kcp-unhealthy-5m
spec:
  clusterName: capi-quickstart
  selector:
    matchLabels:
      cluster.x-k8s.io/control-plane: ""
  checks:
    unhealthyNodeConditions:
    - type: Ready
      status: Unknown
      timeoutSeconds: 300
    - type: Ready
      status: "False"
      timeoutSeconds: 300
  remediation:
    triggerIf:
      unhealthyLessThanOrEqualTo: 100%

Controlling remediation retries

KubeadmControlPlane allows to control how remediation happen by defining an optional remediation; this feature can be used for preventing unnecessary load on infrastructure provider e.g. in case of quota problems,or for allowing the infrastructure provider to stabilize in case of temporary problems.

apiVersion: cluster.x-k8s.io/v1beta2
kind: KubeadmControlPlane
metadata:
  name: my-control-plane
spec:
  ...
  remediation:
    maxRetry: 5
    retryPeriodSeconds: 120 # 2m
    minHealthyPeriodSeconds: 7200 # 2h

maxRetry is the maximum number of retries while attempting to remediate an unhealthy machine. A retry happens when a machine that was created as a replacement for an unhealthy machine also fails. For example, given a control plane with three machines M1, M2, M3:

• M1 become unhealthy; remediation happens, and M1-1 is created as a replacement.
• If M1-1 (replacement of M1) has problems while bootstrapping it will become unhealthy, and then be remediated. This operation is considered a retry - remediation-retry #1.
• If M1-2 (replacement of M1-1) becomes unhealthy, remediation-retry #2 will happen, etc.

A retry will only happen after the retryPeriodSeconds from the previous retry has elapsed. If retryPeriodSeconds is not set (default), a retry will happen immediately.

If a machine is marked as unhealthy after minHealthyPeriodSeconds (default 3600) has passed since the previous remediation this is no longer considered a retry because the new issue is assumed unrelated from the previous one.

If maxRetry is not set (default), remediation will be retried infinitely.

Remediation Short-Circuiting

To ensure that MachineHealthChecks do not perform excessive remediation of Machines, short-circuiting is implemented to prevent further remediation via the remediation.triggerIf field within the MachineHealthCheck spec.

Unhealthy less than or equal to

If the user defines a value for the unhealthyLessThanOrEqualTo field (either an absolute number or a percentage of the total Machines checked by this MachineHealthCheck), before remediating any Machines, the MachineHealthCheck will compare the value of unhealthyLessThanOrEqualTo with the number of Machines it has determined to be unhealthy. If the number of unhealthy Machines exceeds the limit set by unhealthyLessThanOrEqualTo, remediation will not be performed.

With an Absolute Value

If unhealthyLessThanOrEqualTo is set to 2:

• If 2 or fewer nodes are unhealthy, remediation will be performed
• If 3 or more nodes are unhealthy, remediation will not be performed

These values are independent of how many Machines are being checked by the MachineHealthCheck.

With Percentages

If unhealthyLessThanOrEqualTo is set to 40% and there are 25 Machines being checked:

• If 10 or fewer nodes are unhealthy, remediation will be performed
• If 11 or more nodes are unhealthy, remediation will not be performed

If unhealthyLessThanOrEqualTo is set to 40% and there are 6 Machines being checked:

• If 2 or fewer nodes are unhealthy, remediation will be performed
• If 3 or more nodes are unhealthy, remediation will not be performed

Note, when the percentage is not a whole number, the allowed number is rounded down.

Unhealthy in Range

If the user defines a value for the unhealthyInRange field (bracketed values that specify a start and an end value), before remediating any Machines, the MachineHealthCheck will check if the number of Machines it has determined to be unhealthy is within the range specified by unhealthyInRange. If it is not within the range set by unhealthyInRange, remediation will not be performed.

With a range of values

If unhealthyInRange is set to [3-5] and there are 10 Machines being checked:

• If 2 or fewer nodes are unhealthy, remediation will not be performed.
• If 6 or more nodes are unhealthy, remediation will not be performed.
• In all other cases, remediation will be performed.

Note, the above example had 10 machines as sample set. But, this would work the same way for any other number. This is useful for dynamically scaling clusters where the number of machines keep changing frequently.

Skipping Remediation

There are scenarios where remediation for a machine may be undesirable (eg. during cluster migration using clusterctl move). For such cases, MachineHealthCheck skips marking a Machine for remediation if:

• the Machine has the cluster.x-k8s.io/skip-remediation annotation
• the Machine has the cluster.x-k8s.io/paused annotation
• the MachineHealthCheck has the cluster.x-k8s.io/paused annotation
• the Cluster has .spec.paused set to true

Limitations and Caveats of a MachineHealthCheck

Before deploying a MachineHealthCheck, please familiarise yourself with the following limitations and caveats:

• Only Machines owned by a MachineSet or a KubeadmControlPlane can be remediated by a MachineHealthCheck (since a MachineDeployment uses a MachineSet, then this includes Machines that are part of a MachineDeployment)
• Machines managed by a KubeadmControlPlane are remediated according to the delete-and-recreate guidelines described in the KubeadmControlPlane proposal
  • The following rules should be satisfied in order to start remediation of a control plane machine:
    • One of the following apply:
      • The cluster MUST not be initialized yet (the failure happens before KCP reaches the initialized state)
      • The cluster MUST have at least two control plane machines, because this is the smallest cluster size that can be remediated.
    • Previous remediation (delete and re-create) MUST have been completed. This rule prevents KCP from remediating more machines while the replacement for the previous machine is not yet created.
    • The cluster MUST have no machines with a deletion timestamp. This rule prevents KCP taking actions while the cluster is in a transitional state.
    • Remediation MUST preserve etcd quorum. This rule ensures that we will not remove a member that would result in etcd losing a majority of members and thus become unable to field new requests (note: this rule applies only to CP already initialized and with managed etcd)
• If the Node for a Machine is removed from the cluster, a MachineHealthCheck will consider this Machine unhealthy and remediate it immediately
• If no Node joins the cluster for a Machine after the NodeStartupTimeout, the Machine will be remediated
• Important: if the kubelet on the node hosting the etcd leader member is not working, this prevents KCP from doing some checks it is expecting to do on the leader - and specifically on the leader -. This prevents remediation to happen. There are ongoing discussions about how to overcome this limitation in https://github.com/kubernetes-sigs/cluster-api/issues/8465; as of today users facing this situation are recommended to manually forward leadership to another etcd member and manually delete the corresponding machine.

Machine deletion process

Machine deletions occur in various cases, for example:

• Control plane (e.g. KCP) or MachineDeployment rollouts
• Scale downs of MachineDeployments / MachineSets
• Machine remediations
• Machine deletions (e.g. kubectl delete machine)

This page describes how Cluster API deletes Machines.

Machine deletion can be broken down into the following phases:

1. Machine deletion is triggered (i.e. the metadata.deletionTimestamp is set)
2. Machine controller waits until all pre-drain hooks succeeded, if any are registered
   • Pre-drain hooks can be registered by adding annotations with the pre-drain.delete.hook.machine.cluster.x-k8s.io prefix to the Machine object
3. Machine controller checks if the Machine should be drained, drain is skipped if:
   • The Machine has the machine.cluster.x-k8s.io/exclude-node-draining annotation
   • The Machine.spec.nodeDrainTimeout field is set and already expired (unset or 0 means no timeout)
   • The Machine is owned by a KubeadmControlPlane and the pre-terminate hook has been already removed
4. If the Machine should be drained, the Machine controller evicts all relevant Pods from the Node (see details in Node drain)
5. Machine controller checks if we should wait until all volumes are detached, this is skipped if:
   • The Machine has the machine.cluster.x-k8s.io/exclude-wait-for-node-volume-detach annotation
   • The Machine.spec.nodeVolumeDetachTimeout field is set and already expired (unset or 0 means no timeout)
   • The Machine is owned by a KubeadmControlPlane and the pre-terminate hook has been already removed
6. If we should wait for volume detach, the Machine controller waits until Node.status.volumesAttached is empty and there are no more VolumeAttachment objects that indicate that there are still volumes attached to the Node
   • Typically the volumes are getting detached by CSI after the corresponding Pods have been evicted during drain
7. Machine controller waits until all pre-terminate hooks succeeded, if any are registered
   • Pre-terminate hooks can be registered by adding annotations with the pre-terminate.delete.hook.machine.cluster.x-k8s.io prefix to the Machine object
8. Machine controller deletes the InfrastructureMachine object (e.g. DockerMachine) of the Machine and waits until it is gone
9. Machine controller deletes the BootstrapConfig object (e.g. KubeadmConfig) of the machine and waits until it is gone
10. Machine controller deletes the Node object in the workload cluster
    • Node deletion will be retried until either the Node object is gone or Machine.spec.nodeDeletionTimeout is expired (0 means no timeout, but the field defaults to 10s)
    • Note: Nodes are usually also deleted by cloud controller managers, which is why Cluster API per default only tries to delete Nodes for 10s.

Note: There are cases where Node drain, wait for volume detach and Node deletion is skipped. For these please take a look at the implementation of the isDeleteNodeAllowed function.

Node drain

This section describes details of the Node drain process in Cluster API. Cluster API implements Node drain aligned with kubectl drain. One major difference is that the Cluster API controller does not actively wait during Reconcile until all Pods are drained from the Node. Instead it continuously evicts Pods and requeues after 20s until all relevant Pods have been drained from the Node or until the Machine.spec.nodeDrainTimeout is reached (if configured).

Node drain can be broken down into the following phases:

• Node is cordoned (i.e. the Node.spec.unschedulable field is set, which leads to the node.kubernetes.io/unschedulable:NoSchedule taint being added to the Node)
  • This prevents that Pods that already have been evicted are rescheduled to the same Node. Please only tolerate this taint if you know what you are doing! Otherwise it can happen that the Machine controller is stuck continuously evicting the same Pods.
• Machine controller calculates the list of Pods that have to be drained from the Node. Pods can be categorized as follows:
  • Pods that are skipped/ignored during drain:
    • Pods belonging to an existing DaemonSet (orphaned DaemonSet Pods have to be evicted as well)
    • Mirror Pods, i.e. Pods with the kubernetes.io/config.mirror annotation (usually static Pods managed by kubelet, like kube-apiserver)
    • Pods with the cluster.x-k8s.io/drain=skip label
    • Pods that match a MachineDrainRule with behavior Skip
  • Pods that should not be evicted, but we have to wait for their completion:
    • Pods with the cluster.x-k8s.io/drain=wait-completed label
    • Pods that match a MachineDrainRule with behavior WaitCompleted
  • Pods that should be evicted:
    • Pods that match a MachineDrainRule with behavior Drain
    • All Pods not belonging to any of the other categories
• If there are no more Pods that have to be drained Node drain is completed
• Otherwise we have to wait for Pods to complete and/or evict Pods
  • There are various reasons why an eviction could fail:
    • The eviction would violate a PodDisruptionBudget, i.e. not enough Pod replicas would be available if the Pod would be evicted
    • The namespace is in terminating, in this case the kube-controller-manager is responsible for setting the .metadata.deletionTimestamp on the Pod
    • Other errors, e.g. a connection issue when calling the eviction API of the workload cluster
  • Please note that when an eviction goes through, this only means that the .metadata.deletionTimestamp is set on the Pod, but the Pod also has to be terminated and the Pod object has to go away for the drain to complete.
• These steps are repeated every 20s until all relevant Pods have been drained from the Node

Per default all Pods are drained at the same time. But with MachineDrainRules it’s also possible to define a drain order for Pods with behavior Drain (Pods with WaitCompleted have a hard-coded order of 0). The Machine controller will drain Pods in batches based on their order, lowest order first. Pods that don’t match any rule have a default order of 0, so a rule with a negative order will drain its pods before unmatched pods, and a rule with a positive order will drain its pods after.

Example: To drain pods in a specific namespace first and Rook Ceph OSDs last:

apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDrainRule
metadata:
  name: drain-managed-namespaces-first
spec:
  pods:
  - selector:
      matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values: ["my-app-namespace"]
  drain:
    behavior: Drain
    order: -100  # Negative order: drain before default (0)
---
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDrainRule
metadata:
  name: drain-rook-ceph-osd-last
spec:
  pods:
  - selector:
      matchLabels:
        app: rook-ceph-osd
  drain:
    behavior: Drain
    order: 100  # Positive order: drain after default (0)

For more details about MachineDrainRules, please see the corresponding proposal.

Special cases:

• If the Node doesn’t exist anymore, Node drain is entirely skipped
• If the Node is unreachable (i.e. the Node Ready condition is in status Unknown):
  • Pods with .metadata.deletionTimestamp more than 1s in the past are ignored
  • Pod evictions will use 1s GracePeriodSeconds, i.e. the terminationGracePeriodSeconds field from the Pod spec will be ignored.
  • Note: PodDisruptionBudgets are still respected, because both of these changes are only relevant if the call to trigger the Pod eviction goes through. But Pod eviction calls are rejected when PodDisruptionBudgets would be violated by the eviction.

Observability

The drain process can be observed through the DrainingSucceeded condition on the Machine and various logs.

Example condition

To determine which Pods are blocking the drain and why you can take a look at the DrainingSucceeded condition on the Machine, e.g.:

status:
  ...
  conditions:
  ...
  - lastTransitionTime: "2024-08-30T13:36:27Z"
    message: |-
      Drain not completed yet:
      * Pods with deletionTimestamp that still exist: cert-manager/cert-manager-756d54fb98-hcb6k
      * Pods with eviction failed:
        * Cannot evict pod as it would violate the pod's disruption budget. The disruption budget nginx needs 10 healthy pods and has 10 currently: test-namespace/nginx-deployment-6886c85ff7-2jtqm, test-namespace/nginx-deployment-6886c85ff7-7ggsd, test-namespace/nginx-deployment-6886c85ff7-f6z4s, ... (7 more)
    reason: Draining
    severity: Info
    status: "False"
    type: DrainingSucceeded

Example logs

When cordoning the Node:

I0830 12:50:13.961156      17 machine_controller.go:716] "Cordoning Node" ... Node="my-cluster-md-0-wxtcg-mtg57-k9qvz"

When starting the drain:

I0830 12:50:13.961156      17 machine_controller.go:716] "Draining Node" ... Node="my-cluster-md-0-wxtcg-mtg57-k9qvz"

Immediately before Pods are evicted:

I0830 12:52:58.739093      17 drain.go:172] "Drain not completed yet, there are still Pods on the Node that have to be drained" ... Node="my-cluster-md-0-wxtcg-mtg57-ssfg8" podsToTriggerEviction="test-namespace/nginx-deployment-6886c85ff7-4r297, test-namespace/nginx-deployment-6886c85ff7-5gl2h, test-namespace/nginx-deployment-6886c85ff7-64tf9, test-namespace/nginx-deployment-6886c85ff7-9k5gp, test-namespace/nginx-deployment-6886c85ff7-9mdjw, ... (5 more)" podsWithDeletionTimestamp="kube-system/calico-kube-controllers-7dc5458bc6-rdjj4, kube-system/coredns-7db6d8ff4d-9cbhn"

On log level 4 it is possible to observe details of the Pod evictions, e.g.:

I0830 13:29:56.211951      17 drain.go:224] "Evicting Pod" ... Node="my-cluster-2-md-0-wxtcg-mtg57-24lvh" Pod="test-namespace/nginx-deployment-6886c85ff7-77fpw"
I0830 13:29:56.211951      17 drain.go:229] "Pod eviction successfully triggered" ... Node="my-cluster-2-md-0-wxtcg-mtg57-24lvh" Pod="test-namespace/nginx-deployment-6886c85ff7-77fpw"

After Pods have been evicted, either the drain is directly completed:

I0830 13:29:56.235398      17 machine_controller.go:727] "Drain completed, remaining Pods on the Node have been evicted" ... Node="my-cluster-2-md-0-wxtcg-mtg57-24lvh"

or we are requeuing:

I0830 13:29:56.235398      17 machine_controller.go:736] "Drain not completed yet, requeuing in 20s" ... Node="my-cluster-2-md-0-wxtcg-mtg57-24lvh" podsFailedEviction="test-namespace/nginx-deployment-6886c85ff7-77fpw, test-namespace/nginx-deployment-6886c85ff7-8dq4q, test-namespace/nginx-deployment-6886c85ff7-8gjhf, test-namespace/nginx-deployment-6886c85ff7-jznjw, test-namespace/nginx-deployment-6886c85ff7-l5nj8, ... (5 more)" podsWithDeletionTimestamp="kube-system/calico-kube-controllers-7dc5458bc6-rdjj4, kube-system/coredns-7db6d8ff4d-9cbhn"

Eventually the Machine controller should log

I0830 13:29:56.235398      17 machine_controller.go:702] "Drain completed" ... Node="my-cluster-2-md-0-wxtcg-mtg57-24lvh"

If this doesn’t happen, please take a closer at the logs to determine which Pods still have to be evicted or haven’t gone away yet (i.e. deletionTimestamp is set but the Pod objects still exist).

Related documentation

For more information, please see:

• Disruptions: Pod disruption budgets
• Specifying a Disruption Budget for your Application
• API-initiated eviction

Experimental Features

Cluster API now ships with a new experimental package that lives under the exp/ directory. This is a temporary location for features which will be moved to their permanent locations after graduation. Users can experiment with these features by enabling them using feature gates.

Currently Cluster API has the following experimental features:

• ClusterTopology (env var: CLUSTER_TOPOLOGY): ClusterClass
• InPlaceUpdates (env var: EXP_IN_PLACE_UPDATES):
  • Allows users to execute changes on existing machines without deleting the Machine and creating a new one.
  • See the proposal for more details.
• KubeadmBootstrapFormatIgnition (env var: EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION): Ignition
• MachinePool (env var: EXP_MACHINE_POOL): MachinePools
• MachineSetPreflightChecks (env var: EXP_MACHINE_SET_PREFLIGHT_CHECKS): MachineSetPreflightChecks
• MachineTaintPropagation (env var: EXP_MACHINE_TAINT_PROPAGATION): Taint propagation
• PriorityQueue (env var: EXP_PRIORITY_QUEUE): Enables the usage of the controller-runtime PriorityQueue: https://github.com/kubernetes-sigs/controller-runtime/issues/2374
• ReconcilerRateLimiting (env var: EXP_RECONCILER_RATE_LIMITING): Enables reconciler rate-limiting: https://github.com/kubernetes-sigs/cluster-api/issues/13005
  • Note: starting from CAPI v1.12.4 ReconcilerRateLimiting also requires PriorityQueue
• RuntimeSDK (env var: EXP_RUNTIME_SDK): RuntimeSDK

Enabling Experimental Features for Management Clusters Started with clusterctl

Users can enable/disable features by setting OS environment variables before running clusterctl init, e.g.:

export EXP_SOME_FEATURE_NAME=true

clusterctl init --infrastructure vsphere

As an alternative to environment variables, it is also possible to set variables in the clusterctl config file located at $XDG_CONFIG_HOME/cluster-api/clusterctl.yaml, e.g.:

# Values for environment variable substitution
EXP_SOME_FEATURE_NAME: "true"

In case a variable is defined in both the config file and as an OS environment variable, the environment variable takes precedence. For more information on how to set variables for clusterctl, see clusterctl Configuration File

Some features like MachinePools may require infrastructure providers to implement a separate CRD that handles the infrastructure side of the feature too. For such a feature to work, infrastructure providers should also enable their controllers if it is implemented as a feature. If it is not implemented as a feature, no additional step is necessary. As an example, Cluster API Provider Azure (CAPZ) has support for MachinePool through the infrastructure type AzureMachinePool.

Enabling Experimental Features for e2e Tests

One way is to set experimental variables on the clusterctl config file. For CAPI, these configs are under ./test/e2e/config/… such as docker.yaml:

variables:
  CLUSTER_TOPOLOGY: "true"
  EXP_RUNTIME_SDK: "true"
  EXP_MACHINE_SET_PREFLIGHT_CHECKS: "true"

Another way is to set them as environmental variables before running e2e tests.

Enabling Experimental Features on Tilt

On development environments started with Tilt, features can be enabled by setting the feature variables in kustomize_substitutions, e.g.:

kustomize_substitutions:
  CLUSTER_TOPOLOGY: 'true'
  EXP_RUNTIME_SDK: 'true'
  EXP_MACHINE_SET_PREFLIGHT_CHECKS: 'true'

For more details on setting up a development environment with tilt, see Developing Cluster API with Tilt

Enabling Experimental Features on Existing Management Clusters

To enable/disable features on existing management clusters, users can edit the corresponding controller manager deployments, which will then trigger a restart with the requested features. E.g. for the CAPI controller manager deployment:

kubectl edit -n capi-system deployment.apps/capi-controller-manager

// Enable/disable available features by modifying Args below.
    Args:
      --leader-elect
      --feature-gates=MachinePool=true,ClusterResourceSet=true

Similarly, to validate if a particular feature is enabled, see the arguments by issuing:

kubectl describe -n capi-system deployment.apps/capi-controller-manager

Following controller manager deployments have to be edited in order to enable/disable their respective experimental features:

• MachinePools:
  • CAPI.
  • CABPK.
  • CAPD. Other Infrastructure Providers might also require this. Please consult the docs of the concrete Infrastructure Provider regarding this.
• ClusterClass:
  • CAPI.
  • KCP.
  • CAPD. Other Infrastructure Providers might also require this. Please consult the docs of the concrete Infrastructure Provider regarding this.
• Ignition Bootstrap configuration:
  • CABPK.
  • KCP.
• Runtime SDK:
  • CAPI.

Active Experimental Features

• MachinePools
• ClusterClass
• Ignition Bootstrap configuration
• Runtime SDK

Warning: Experimental features are unreliable, i.e., some may one day be promoted to the main repository, or they may be modified arbitrarily or even disappear altogether. In short, they are not subject to any compatibility or deprecation promise.

Experimental Feature: MachinePool (beta)

Feature gate name: MachinePool

Variable name to enable/disable the feature gate: EXP_MACHINE_POOL

Table of Contents

• Introduction
• What is a MachinePool?
• Why MachinePool?
• When to use MachinePool vs MachineDeployment
• Enabling MachinePool
• MachinePool provider implementations
• Additional Resources

Introduction

Cluster API (CAPI) manages Kubernetes worker nodes primarily through Machine, MachineSet, and MachineDeployment objects. These primitives manage nodes individually (Machines), and have served well across a wide variety of providers.

However, many infrastructure providers already offer first-class abstractions for groups of compute instances (AWS: Auto Scaling Groups (ASG), Azure: Virtual Machine Scale Sets (VMSS), or GCP: Managed Instance Groups (MIG)). These primitives natively support scaling, rolling upgrades, and health management.

MachinePool brings these provider features into Cluster API by introducing a higher-level abstraction for managing a group of machines as a single unit.

What is a MachinePool?

A MachinePool is a Cluster API resource representing a group of worker nodes. Instead of reconciling each machine individually, CAPI delegates lifecycle management to the infrastructure provider.

• MachinePool (core API): defines desired state (replicas, Kubernetes version, bootstrap template, infrastructure reference).
• InfrastructureMachinePool (provider API): provides an implementation that backs a pool. A provider may offer more than one type depending on how it is managed. For example:
  • AWSMachinePool: self-managed ASG
  • AWSManagedMachinePool: EKS managed node group
  • AzureMachinePool: VM Scale Set
  • AzureManagedMachinePool: AKS managed node pool
  • GCPManagedMachinePool: GKE managed node pool
  • OCIManagedMachinePool: OKE managed node pool
  • ScalewayManagedMachinePool: Scaleway Kapsule node pool
• Bootstrap configuration: still applies (e.g., kubeadm configs), ensuring that new nodes join the cluster with the correct setup.

The MachinePool controller coordinates between the Cluster API core and provider-specific implementations:

• Reconciles desired replicas with the infrastructure pool.
• Matches provider IDs from the infrastructure resource with Kubernetes Nodes in the workload cluster.
• Updates MachinePool status (ready replicas, conditions, etc.)

Why MachinePool?

Leverage provider primitives

Most cloud providers already manage scaling, instance replacement, and health monitoring at the group level. MachinePool lets CAPI delegate lifecycle operations instead of duplicating that logic.

Example:

• AWS Auto Scaling Groups replace failed nodes automatically.
• Azure VM Scale Sets support rolling upgrades with configurable surge/availability strategies.

Simplify upgrades and scaling

Upgrades and scaling events are managed at the pool level:

• Update Kubernetes version or bootstrap template → cloud provider handles rolling replacement.
• Scale up/down replicas → provider adjusts capacity.

This provides more predictable, cloud-native semantics compared to reconciling many individual Machine objects.

Autoscaling integration

MachinePool integrates with the Cluster Autoscaler in the same way that MachineDeployments do. In practice, the autoscaler treats a MachinePool as a node group, enabling scale-up and scale-down decisions based on cluster load.

Tradeoffs and limitations

While powerful, MachinePool comes with tradeoffs:

• Infrastructure provider complexity: requires infrastructure providers to implement and maintain an InfrastructureMachinePool type.
• Less per-machine granularity: you cannot configure each node individually; the pool defines a shared template.

  Note: While this is typically true, certain cloud providers do offer flexibility. Example: AWS allows AWSMachinepool.spec.mixedInstancesPolicy.instancesDistribution while Azure allows AzureMachinePool.spec.orchestrationMode.

• Complex reconciliation: node-to-providerID matching introduces edge cases (delays, inconsistent states).
• Draining: The cloud resources for MachinePool may not necessarily support draining of Kubernetes worker nodes. For example, with an AWSMachinePool, AWS would normally terminate instances as quickly as possible. To solve this, tools like aws-node-termination-handler combined with ASG lifecycle hooks (defined in AWSMachine.spec.lifecycleHooks) must be installed, and is not a built-in feature of the infrastructure provider (CAPA in this example).
• Maturity: The MachinePool API is still considered experimental/beta.

When to use MachinePool vs MachineDeployment

Both MachineDeployment and MachinePool are valid options for managing worker nodes in Cluster API. The right choice depends on your infrastructure provider’s capabilities and your operational requirements.

Use MachinePool when:

• Cloud provider supports scaling group primitives: AWS Auto Scaling Groups, Azure Virtual Machine Scale Sets, GCP Managed Instance Groups, OCI Compute Instances, Scaleway Kapsule. These resources natively handle scaling, rolling upgrades, and health checks.
• You want to leverage cloud provider-level features: MachinePool enables direct use of cloud-native upgrade strategies (e.g., surge, maxUnavailable) and autoscaling behaviors.

Use MachineDeployment when:

• The provider does not support scaling groups: Common in environments such as bare metal, vSphere, or Docker.
• You need fine-grained per-machine control: MachineDeployments allow unique bootstrap configurations, labels, and taints across different MachineSets.
• You prefer maturity and portability: MachineDeployment is stable, GA, and supported across all providers. MachinePool remains experimental in some implementations.

Enabling MachinePool

Starting from Cluster API v1.7, MachinePool is enabled by default. No additional configuration is needed.

For Cluster API versions prior to v1.7, you need to set the EXP_MACHINE_POOL environment variable:

export EXP_MACHINE_POOL=true
clusterctl init

Or when upgrading an existing management cluster:

export EXP_MACHINE_POOL=true
clusterctl upgrade

MachinePool provider implementations

The following Cluster API infrastructure providers have implemented support for MachinePools:

Providers may support the deletion of single machine pool Machine objects. That allows, for example, using MachineHealthCheck to remediate machines that became unhealthy (requires this PR to be merged and released).

Additional Resources

• Design Document: MachinePool CAEP
• Developer Documentation: MachinePool Controller

Experimental Feature: MachineSetPreflightChecks (beta)

The MachineSetPreflightChecks feature can provide additional safety while creating new Machines and remediating existing unhealthy Machines of a MachineSet.

When a MachineSet creates machines under certain circumstances, the operation fails or leads to a new machine that will be deleted and recreated in a short timeframe, leading to unwanted Machine churn. Some of these circumstances include, but not limited to, creating a new Machine when Kubernetes version skew could be violated or joining a Machine when the Control Plane is upgrading leading to failure because of mixed kube-apiserver version or due to the cluster load balancer delays in adapting to the changes.

Enabling MachineSetPreflightChecks provides safety in such circumstances by making sure that a Machine is only created when it is safe to do so.

Feature gate name: MachineSetPreflightChecks

Variable name to enable/disable the feature gate: EXP_MACHINE_SET_PREFLIGHT_CHECKS

Supported PreflightChecks

ControlPlaneIsStable

• This preflight check ensures that the ControlPlane is currently stable i.e. the ControlPlane is currently neither provisioning, upgrading.
• For Clusters with a managed topology it also checks if a control plane upgrade is pending.
• This preflight check is only performed if:
  • The Cluster uses a ControlPlane provider.
  • ControlPlane version is defined (ControlPlane.spec.version is set).

KubernetesVersionSkew

• This preflight check ensures that the MachineSet and the ControlPlane conform to the Kubernetes version skew.
• This preflight check is only performed if:
  • The Cluster uses a ControlPlane provider.
  • ControlPlane version is defined (ControlPlane.spec.version is set).
  • MachineSet version is defined (MachineSet.spec.template.spec.version is set).

KubeadmVersionSkew

• This preflight check ensures that the MachineSet and the ControlPlane conform to the kubeadm version skew.
• This preflight check is only performed if:
  • The Cluster uses a ControlPlane provider.
  • ControlPlane version is defined (ControlPlane.spec.version is set).
  • MachineSet version is defined (MachineSet.spec.template.spec.version is set).
  • MachineSet uses the Kubeadm Bootstrap provider.

ControlPlaneVersionSkew

• This preflight check ensures that the MachineSet and the ControlPlane have the same version. The idea behind this check is that it doesn’t make sense to create a Machine with an old version, if we already know based on the control plane version that the Machine has to be replaced soon.
• This preflight check is only performed if:
  • The Cluster has a managed topology
  • The Cluster uses a ControlPlane provider.
  • ControlPlane version is defined (ControlPlane.spec.version is set).
  • MachineSet version is defined (MachineSet.spec.template.spec.version is set).

Configuring MachineSet PreflightChecks

Per default all preflight checks are enabled for all MachineSets including new and existing MachineSets. The enabled preflight checks can be overwritten with the --machineset-preflight-checks command-line flag.

It is also possible to opt-out of one or all of the preflight checks on a per MachineSet basis by specifying a comma-separated list of the preflight checks via the machineset.cluster.x-k8s.io/skip-preflight-checks annotation on the MachineSet or on the corresponding BootstrapConfigTemplate (annotation on the MachineSet has higher priority).

Examples:

• To opt out of all the preflight checks set the machineset.cluster.x-k8s.io/skip-preflight-checks: All annotation.
• To opt out of the ControlPlaneIsStable preflight check set the machineset.cluster.x-k8s.io/skip-preflight-checks: ControlPlaneIsStable annotation.
• To opt out of multiple preflight checks set the machineset.cluster.x-k8s.io/skip-preflight-checks: ControlPlaneIsStable,KubernetesVersionSkew annotation.

Experimental Feature: ClusterClass (alpha)

The ClusterClass feature introduces a new way to create clusters which reduces boilerplate and enables flexible and powerful customization of clusters. ClusterClass is a powerful abstraction implemented on top of existing interfaces and offers a set of tools and operations to streamline cluster lifecycle management while maintaining the same underlying API.

Feature gate name: ClusterTopology

Variable name to enable/disable the feature gate: CLUSTER_TOPOLOGY

Additional documentation:

• Background information: ClusterClass and Managed Topologies CAEP
• For ClusterClass authors:
  • Writing a ClusterClass
  • Changing a ClusterClass
  • Publishing a ClusterClass for clusterctl usage: clusterctl Provider contract
• For Cluster operators:
  • Creating a Cluster: Quick Start guide Please note that the experience for creating a Cluster using ClusterClass is very similar to the one for creating a standalone Cluster. Infrastructure providers supporting ClusterClass provide Cluster templates leveraging this feature (e.g the Docker infrastructure provider has a development-topology template).
  • Operating a managed Cluster

Writing a ClusterClass

A ClusterClass becomes more useful and valuable when it can be used to create many Cluster of a similar shape. The goal of this document is to explain how ClusterClasses can be written in a way that they are flexible enough to be used in as many Clusters as possible by supporting variants of the same base Cluster shape.

Table of Contents

• Basic ClusterClass
• ClusterClass with MachineHealthChecks
• ClusterClass with patches
• ClusterClass with custom naming strategies
  • Defining a custom naming strategy for ControlPlane objects
  • Defining a custom naming strategy for MachineDeployment objects
  • Defining a custom naming strategy for MachinePool objects
• Advanced features of ClusterClass with patches
  • MachineDeployment variable overrides
  • Builtin variables
  • Complex variable types
  • Using variable values in JSON patches
  • Optional patches
  • Version-aware patches
• JSON patches tips & tricks

Basic ClusterClass

The following example shows a basic ClusterClass. It contains templates to shape the control plane, infrastructure and workers of a Cluster. When a Cluster is using this ClusterClass, the templates are used to generate the objects of the managed topology of the Cluster.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  controlPlane:
    templateRef:
      apiVersion: controlplane.cluster.x-k8s.io/v1beta2
      kind: KubeadmControlPlaneTemplate
      name: docker-clusterclass-v0.1.0
    machineInfrastructure:
      templateRef:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: DockerMachineTemplate
        name: docker-clusterclass-v0.1.0
  infrastructure:
    templateRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      kind: DockerClusterTemplate
      name: docker-clusterclass-v0.1.0-control-plane
  workers:
    machineDeployments:
    - class: default-worker
      bootstrap:
        templateRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
          kind: KubeadmConfigTemplate
          name: docker-clusterclass-v0.1.0-default-worker
      infrastructure:
        templateRef:
          apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
          kind: DockerMachineTemplate
          name: docker-clusterclass-v0.1.0-default-worker

The following example shows a Cluster using this ClusterClass. In this case a KubeadmControlPlane with the corresponding DockerMachineTemplate, a DockerCluster and a MachineDeployment with the corresponding KubeadmConfigTemplate and DockerMachineTemplate will be created. This basic ClusterClass is already very flexible. Via the topology on the Cluster the following can be configured:

• .spec.topology.version: the Kubernetes version of the Cluster
• .spec.topology.controlPlane: ControlPlane replicas and their metadata
• .spec.topology.workers: MachineDeployments and their replicas, metadata and failure domain

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: my-docker-cluster
spec:
  topology:
    classRef:
      name: docker-clusterclass-v0.1.0
    version: v1.22.4
    controlPlane:
      replicas: 3
      metadata:
        labels:
          cpLabel: cpLabelValue 
        annotations:
          cpAnnotation: cpAnnotationValue
    workers:
      machineDeployments:
      - class: default-worker
        name: md-0
        replicas: 4
        metadata:
          labels:
            mdLabel: mdLabelValue
          annotations:
            mdAnnotation: mdAnnotationValue
        failureDomain: region

Best practices:

• The ClusterClass name should be generic enough to make sense across multiple clusters, i.e. a name which corresponds to a single Cluster, e.g. “my-cluster”, is not recommended.
• Try to keep the ClusterClass names short and consistent (if you publish multiple ClusterClasses).
• As a ClusterClass usually evolves over time and you might want to rebase Clusters from one version of a ClusterClass to another, consider including a version suffix in the ClusterClass name. For more information about changing a ClusterClass please see: Changing a ClusterClass.
• Prefix the templates used in a ClusterClass with the name of the ClusterClass.
• Don’t reuse the same template in multiple ClusterClasses. This is automatically taken care of by prefixing the templates with the name of the ClusterClass.

ClusterClass with MachinePools

ClusterClass also supports MachinePool workers. They work very similar to MachineDeployments. MachinePools can be specified in the ClusterClass template under the workers section like so:

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  workers:
    machinePools:
    - class: default-worker
      bootstrap:
        templateRef:
          apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
          kind: KubeadmConfigTemplate
          name: quick-start-default-worker-bootstraptemplate
      infrastructure:
        templateRef:
          apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
          kind: DockerMachinePoolTemplate
          name: quick-start-default-worker-machinepooltemplate

They can then be similarly defined as workers in the cluster template like so:

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: my-docker-cluster
spec:
  topology:
    workers:
      machinePools:
      - class: default-worker
        name: mp-0
        replicas: 4
        metadata:
          labels:
            mpLabel: mpLabelValue
          annotations:
            mpAnnotation: mpAnnotationValue
        failureDomain: region

ClusterClass with MachineHealthChecks

MachineHealthChecks can be configured in the ClusterClass for the control plane and for a MachineDeployment class. The following configuration makes sure a MachineHealthCheck is created for the control plane and for every MachineDeployment using the default-worker class.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  controlPlane:
    ...
    healthCheck:
      checks:
        nodeStartupTimeoutSeconds: 900
        unhealthyNodeConditions:
        - type: Ready
          status: Unknown
          timeoutSeconds: 300
        - type: Ready
          status: "False"
          timeoutSeconds: 300
        unhealthyMachineConditions:
        - type: "NodeReady"
          status: Unknown
          timeoutSeconds: 1800
        - type: "InfrastructureReady"
          status: "False"
          timeoutSeconds: 1800
      remediation:
        triggerIf:
          unhealthyLessThanOrEqualTo: 33%
  workers:
    machineDeployments:
    - class: default-worker
      ...
      healthCheck:
        checks:
          nodeStartupTimeoutSeconds: 600
          unhealthyNodeConditions:
          - type: Ready
            status: Unknown
            timeoutSeconds: 300
          - type: Ready
            status: "False"
            timeoutSeconds: 300
          unhealthyMachineConditions:
          - type: NodeReady
            status: Unknown
            timeoutSeconds: 1800
          - type: InfrastructureReady
            status: "False"
            timeoutSeconds: 1800
        remediation:
          triggerIf:
            unhealthyInRange: "[0-2]"

ClusterClass with patches

As shown above, basic ClusterClasses are already very powerful. But there are cases where more powerful mechanisms are required. Let’s assume you want to manage multiple Clusters with the same ClusterClass, but they require different values for a field in one of the referenced templates of a ClusterClass.

A concrete example would be to deploy Clusters with different registries. In this case, every cluster needs a Cluster-specific value for .spec.kubeadmConfigSpec.clusterConfiguration.imageRepository in KubeadmControlPlane. Use cases like this can be implemented with ClusterClass patches.

Defining variables in the ClusterClass

The following example shows how variables can be defined in the ClusterClass. A variable definition specifies the name and the schema of a variable and if it is required. The schema defines how a variable is defaulted and validated. It supports a subset of the schema of CRDs. For more information please see the godoc.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  ...
  variables:
  - name: imageRepository
    required: true
    schema:
      openAPIV3Schema:
        type: string
        description: ImageRepository is the container registry to pull images from.
        default: registry.k8s.io
        example: registry.k8s.io

Defining patches in the ClusterClass

The variable can then be used in a patch to set a field on a template referenced in the ClusterClass. The selector specifies on which template the patch should be applied. jsonPatches specifies which JSON patches should be applied to that template. In this case we set the imageRepository field of the KubeadmControlPlaneTemplate to the value of the variable imageRepository. For more information please see the godoc.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  ...
  patches:
  - name: imageRepository
    definitions:
    - selector:
        apiVersion: controlplane.cluster.x-k8s.io/v1beta2
        kind: KubeadmControlPlaneTemplate
        matchResources:
          controlPlane: true
      jsonPatches:
      - op: add
        path: /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/imageRepository
        valueFrom:
          variable: imageRepository

Setting variable values in the Cluster

After creating a ClusterClass with a variable definition, the user can now provide a value for the variable in the Cluster as in the example below.

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: my-docker-cluster
spec:
  topology:
    ...
    variables:
    - name: imageRepository
      value: my.custom.registry

ClusterClass with custom naming strategies

The controller needs to generate names for new objects when a Cluster is getting created from a ClusterClass. These names have to be unique for each namespace. The naming strategy enables this by concatenating the cluster name with a random suffix.

It is possible to provide a custom template for the name generation of ControlPlane, MachineDeployment and MachinePool objects.

The generated names must comply with the RFC 1123 standard.

Defining a custom naming strategy for ControlPlane objects

The naming strategy for ControlPlane supports the following properties:

• template: Custom template which is used when generating the name of the ControlPlane object.

The following variables can be referenced in templates:

• .cluster.name: The name of the cluster object.
• .random: A random alphanumeric string, without vowels, of length 5.

Example which would match the default behavior:

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  controlPlane:
    ...
    naming:
      template: "{{ .cluster.name }}-{{ .random }}"
  ...

Defining a custom naming strategy for MachineDeployment objects

The naming strategy for MachineDeployments supports the following properties:

• template: Custom template which is used when generating the name of the MachineDeployment object.

The following variables can be referenced in templates:

• .cluster.name: The name of the cluster object.
• .random: A random alphanumeric string, without vowels, of length 5.
• .machineDeployment.topologyName: The name of the MachineDeployment topology (Cluster.spec.topology.workers.machineDeployments[].name)

Example which would match the default behavior:

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  controlPlane:
    ...
  workers:
    machineDeployments:
    - class: default-worker
      ...
      naming:
        template: "{{ .cluster.name }}-{{ .machineDeployment.topologyName }}-{{ .random }}"

Defining a custom naming strategy for MachinePool objects

The naming strategy for MachinePools supports the following properties:

• template: Custom template which is used when generating the name of the MachinePool object.

The following variables can be referenced in templates:

• .cluster.name: The name of the cluster object.
• .random: A random alphanumeric string, without vowels, of length 5.
• .machinePool.topologyName: The name of the MachinePool topology (Cluster.spec.topology.workers.machinePools[].name).

Example which would match the default behavior:

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  controlPlane:
    ...
  workers:
    machinePools:
    - class: default-worker
      ...
      naming:
        template: "{{ .cluster.name }}-{{ .machinePool.topologyName }}-{{ .random }}"

Defining a custom namespace for ClusterClass object

As a user, I may need to create a Cluster from a ClusterClass object that exists only in a different namespace. To uniquely identify the ClusterClass, a NamespacedName ref is constructed from combination of:

• cluster.spec.topology.classNamespace - namespace of the ClusterClass object.
• cluster.spec.topology.class - name of the ClusterClass object.

Example of the Cluster object with the name/namespace reference:

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: my-docker-cluster
  namespace: default
spec:
  topology:
    classRef:
      name: docker-clusterclass-v0.1.0
      namespace: default
    version: v1.22.4
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
      - class: default-worker
        name: md-0
        replicas: 4
        failureDomain: region

Securing cross-namespace reference to the ClusterClass

It is often desirable to restrict free cross-namespace ClusterClass access for the Cluster object. This can be implemented by defining a ValidatingAdmissionPolicy on the Cluster object.

An example of such policy may be:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: "cluster-class-ref.cluster.x-k8s.io"
spec:
  failurePolicy: Fail
  paramKind:
    apiVersion: v1
    kind: Secret
  matchConstraints:
    resourceRules:
    - apiGroups:   ["cluster.x-k8s.io"]
      apiVersions: ["v1beta2"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["clusters"]
  validations:
    - expression: "!has(object.spec.topology.classRef.namespace) || object.spec.topology.classRef.namespace in params.data"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: "cluster-class-ref-binding.cluster.x-k8s.io"
spec:
  policyName: "cluster-class-ref.cluster.x-k8s.io"
  validationActions: [Deny]
  paramRef:
    name: "allowed-namespaces.cluster-class-ref.cluster.x-k8s.io"
    namespace: "default"
    parameterNotFoundAction: Deny
---
apiVersion: v1
kind: Secret
metadata:
  name: "allowed-namespaces.cluster-class-ref.cluster.x-k8s.io"
  namespace: "default"
data:
  default: ""

Advanced features of ClusterClass with patches

This section will explain more advanced features of ClusterClass patches.

MachineDeployment and MachinePool variable overrides

If you want to use many variations of MachineDeployments in Clusters, you can either define a MachineDeployment class for every variation or you can define patches and variables to make a single MachineDeployment class more flexible. The same applies for MachinePools.

In the following example we make the instanceType of a AWSMachineTemplate customizable. First we define the workerMachineType variable and the corresponding patch:

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: aws-clusterclass-v0.1.0
spec:
  ...
  variables:
  - name: workerMachineType
    required: true
    schema:
      openAPIV3Schema:
        type: string
        default: t3.large
  patches:
  - name: workerMachineType
    definitions:
    - selector:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: AWSMachineTemplate
        matchResources:
          machineDeploymentClass:
            names:
            - default-worker
      jsonPatches:
      - op: add
        path: /spec/template/spec/instanceType
        valueFrom:
          variable: workerMachineType
---
apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
kind: AWSMachineTemplate
metadata:
  name: aws-clusterclass-v0.1.0-default-worker
spec:
  template:
    spec:
      # instanceType: workerMachineType will be set by the patch.
      iamInstanceProfile: "nodes.cluster-api-provider-aws.sigs.k8s.io"
---
...

In the Cluster resource the workerMachineType variable can then be set cluster-wide and it can also be overridden for an individual MachineDeployment or MachinePool.

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: my-aws-cluster
spec:
  ...
  topology:
    classRef:
      name: aws-clusterclass-v0.1.0
    version: v1.22.0
    controlPlane:
      replicas: 3
    workers:
      machineDeployments:
      - class: "default-worker"
        name: "md-small-workers"
        replicas: 3
        variables:
          overrides:
          # Overrides the cluster-wide value with t3.small.
          - name: workerMachineType
            value: t3.small
      # Uses the cluster-wide value t3.large.
      - class: "default-worker"
        name: "md-large-workers"
        replicas: 3
    variables:
    - name: workerMachineType
      value: t3.large

Builtin variables

In addition to variables specified in the ClusterClass, the following builtin variables can be referenced in patches:

• builtin.cluster.{name,namespace,uid,metadata.labels,metadata.annotations}
• builtin.cluster.topology.{version,classRef.name,classRef.namespace,class,classNamespace}
  • Note: class and classNamespace are deprecated and will be removed with the next apiVersion.
• builtin.cluster.network.{serviceDomain,services,pods}
• builtin.controlPlane.{replicas,version,name,metadata.labels,metadata.annotations}
  • Please note, these variables are only available when patching control plane or control plane machine templates.
• builtin.controlPlane.machineTemplate.infrastructureRef.name
  • Please note, these variables are only available when using a control plane with machines and when patching control plane or control plane machine templates.
• builtin.machineDeployment.{replicas,version,class,name,topologyName,metadata.labels,metadata.annotations}
  • Please note, these variables are only available when patching the templates of a MachineDeployment and contain the values of the current MachineDeployment topology.
• builtin.machineDeployment.{infrastructureRef.name,bootstrap.configRef.name}
  • Please note, these variables are only available when patching the templates of a MachineDeployment and contain the values of the current MachineDeployment topology.
• builtin.machinePool.{replicas,version,class,name,topologyName,metadata.labels,metadata.annotations}
  • Please note, these variables are only available when patching the templates of a MachinePool and contain the values of the current MachinePool topology.
• builtin.machinePool.{infrastructureRef.name,bootstrap.configRef.name}
  • Please note, these variables are only available when patching the templates of a MachinePool and contain the values of the current MachinePool topology.

Builtin variables can be referenced just like regular variables, e.g.:

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  ...
  patches:
  - name: clusterName
    definitions:
    - selector:
      ...
      jsonPatches:
      - op: add
        path: /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/controllerManager/extraArgs/cluster-name
        valueFrom:
          variable: builtin.cluster.name

Tips & Tricks

Builtin variables can be used to dynamically calculate image names. The version used in the patch will always be the same as the one we set in the corresponding MachineDeployment or MachinePool (works the same way with .builtin.controlPlane.version).

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  ...
  patches:
  - name: customImage
    description: "Sets the container image that is used for running dockerMachines."
    definitions:
    - selector:
        apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
        kind: DockerMachineTemplate
        matchResources:
          machineDeploymentClass:
            names:
            - default-worker
      jsonPatches:
      - op: add
        path: /spec/template/spec/customImage
        valueFrom:
          template: |
            kindest/node:{{ .builtin.machineDeployment.version }}

Complex variable types

Variables can also be objects, maps and arrays. An object is specified with the type object and by the schemas of the fields of the object. A map is specified with the type object and the schema of the map values. An array is specified via the type array and the schema of the array items.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  ...
  variables:
  - name: httpProxy
    schema:
      openAPIV3Schema:
        type: object
        properties: 
          # Schema of the url field.
          url: 
            type: string
          # Schema of the noProxy field.
          noProxy:
            type: string
  - name: mdConfig
    schema:
      openAPIV3Schema:
        type: object
        additionalProperties:
          # Schema of the map values.
          type: object
          properties:
            osImage:
              type: string
  - name: dnsServers
    schema:
      openAPIV3Schema:
        type: array
        items:
          # Schema of the array items.
          type: string

Objects, maps and arrays can be used in patches either directly by referencing the variable name, or by accessing individual fields. For example:

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  ...
  jsonPatches:
  - op: add
    path: /spec/template/spec/httpProxy/url
    valueFrom:
      # Use the url field of the httpProxy variable.
      variable: httpProxy.url
  - op: add
    path: /spec/template/spec/customImage
    valueFrom:
      # Use the osImage field of the mdConfig variable for the current MD class.
      template: "{{ (index .mdConfig .builtin.machineDeployment.class).osImage }}"
  - op: add
    path: /spec/template/spec/dnsServers
    valueFrom:
      # Use the entire dnsServers array.
      variable: dnsServers
  - op: add
    path: /spec/template/spec/dnsServer
    valueFrom:
      # Use the first item of the dnsServers array.
      variable: dnsServers[0]

Tips & Tricks

Complex variables can be used to make references in templates configurable, e.g. the identityRef used in AzureCluster. Of course it’s also possible to only make the name of the reference configurable, including restricting the valid values to a pre-defined enum.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: azure-clusterclass-v0.1.0
spec:
  ...
  variables:
  - name: clusterIdentityRef
    schema:
      openAPIV3Schema:
        type: object
        properties:
          kind:
            type: string
          name:
            type: string

Even if OpenAPI schema allows defining free form objects, e.g.

variables:
  - name: freeFormObject
    schema:
      openAPIV3Schema:
        type: object

User should be aware that the lack of the validation of users provided data could lead to problems when those values are used in patch or when the generated templates are created (see e.g. 6135).

As a consequence we recommend avoiding this practice while we are considering alternatives to make it explicit for the ClusterClass authors to opt in this feature, thus accepting the implied risks.

Using variable values in JSON patches

We already saw above that it’s possible to use variable values in JSON patches. It’s also possible to calculate values via Go templating or to use hard-coded values.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  ...
  patches:
  - name: etcdImageTag
    definitions:
    - selector:
      ...
      jsonPatches:
      - op: add
        path: /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/etcd
        valueFrom:
          # This template is first rendered with Go templating, then parsed by 
          # a YAML/JSON parser and then used as value of the JSON patch.
          # For example, if the variable etcdImageTag is set to `3.5.1-0` the 
          # .../clusterConfiguration/etcd field will be set to:
          # {"local": {"imageTag": "3.5.1-0"}}
          template: |
            local:
              imageTag: {{ .etcdImageTag }}
  - name: imageRepository
    definitions:
    - selector:
      ...
      jsonPatches:
      - op: add
        path: /spec/template/spec/kubeadmConfigSpec/clusterConfiguration/imageRepository
        # This hard-coded value is used directly as value of the JSON patch.
        value: "my.custom.registry"

Tips & Tricks

Templates can be used to implement defaulting behavior during JSON patch value calculation. This can be used if the simple constant default value which can be specified in the schema is not enough.

        valueFrom:
          # If .vnetName is set, it is used. Otherwise, we will use `{{.builtin.cluster.name}}-vnet`.  
          template: "{{ if .vnetName }}{{.vnetName}}{{else}}{{.builtin.cluster.name}}-vnet{{end}}"

When writing templates, a subset of functions from the Sprig library can be used to write expressions, e.g., {{ .name | upper }}. Only functions that are guaranteed to evaluate to the same result for a given input are allowed (e.g. upper or max can be used, while now or randAlpha cannot be used).

Optional patches

Patches can also be conditionally enabled. This can be done by configuring a Go template via enabledIf. The patch is then only applied if the Go template evaluates to true. In the following example the httpProxy patch is only applied if the httpProxy variable is set (and not empty).

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: docker-clusterclass-v0.1.0
spec:
  ...
  variables:
  - name: httpProxy
    schema:
      openAPIV3Schema:
        type: string
  patches:
  - name: httpProxy
    enabledIf: "{{ if .httpProxy }}true{{end}}"
    definitions:
    ...  

Tips & Tricks:

Hard-coded values can be used to test the impact of a patch during development, gradually roll out patches, etc. .

    enabledIf: false

A boolean variable can be used to enable/disable a patch (or “feature”). This can have opt-in or opt-out behavior depending on the default value of the variable.

    enabledIf: "{{ .httpProxyEnabled }}"

Of course the same is possible by adding a boolean variable to a configuration object.

    enabledIf: "{{ .httpProxy.enabled }}"

Builtin variables can be leveraged to apply a patch only for a specific Kubernetes version.

    enabledIf: '{{ semverCompare "1.21.1" .builtin.controlPlane.version }}'

With semverCompare and coalesce a feature can be enabled in newer versions of Kubernetes for both KubeadmConfigTemplate and KubeadmControlPlane.

    enabledIf: '{{ semverCompare "^1.22.0" (coalesce .builtin.controlPlane.version .builtin.machineDeployment.version )}}'

Version-aware patches

In some cases the ClusterClass authors want a patch to be computed according to the Kubernetes version in use.

While this is not a problem “per se” and it does not differ from writing any other patch, it is important to keep in mind that there could be different Kubernetes version in a Cluster at any time, all of them accessible via built in variables:

• builtin.cluster.topology.version defines the Kubernetes version from cluster.topology, and it acts as the desired Kubernetes version for the entire cluster. However, during an upgrade workflow it could happen that some objects in the Cluster are still at the older version.
• builtin.controlPlane.version, represent the desired version for the control plane object; usually this version changes immediately after cluster.topology.version is updated (unless there are other operations in progress preventing the upgrade to start).
• builtin.machineDeployment.version, represent the desired version for each specific MachineDeployment object; this version changes only after the upgrade for the control plane is completed, and in case of many MachineDeployments in the same cluster, they are upgraded sequentially.
• builtin.machinePool.version, represent the desired version for each specific MachinePool object; this version changes only after the upgrade for the control plane is completed, and in case of many MachinePools in the same cluster, they are upgraded sequentially.

This info should provide the bases for developing version-aware patches, allowing the patch author to determine when a patch should adapt to the new Kubernetes version by choosing one of the above variables. In practice the following rules applies to the most common use cases:

• When developing a version-aware patch for the control plane, builtin.controlPlane.version must be used.
• When developing a version-aware patch for MachineDeployments, builtin.machineDeployment.version must be used.
• When developing a version-aware patch for MachinePools, builtin.machinePool.version must be used.

Tips & Tricks:

Sometimes users need to define variables to be used by version-aware patches, and in this case it is important to keep in mind that there could be different Kubernetes versions in a Cluster at any time.

A simple approach to solve this problem is to define a map of version-aware variables, with the key of each item being the Kubernetes version. Patch could then use the proper builtin variables as a lookup entry to fetch the corresponding values for the Kubernetes version in use by each object.

JSON patches tips & tricks

JSON patches specification RFC6902 requires that the target of add operation must exist.

As a consequence ClusterClass authors should pay special attention when the following conditions apply in order to prevent errors when a patch is applied:

• the patch tries to add a value to an array (which is a slice in the corresponding go struct)
• the slice was defined with omitempty
• the slice currently does not exist

A workaround in this particular case is to create the array in the patch instead of adding to the non-existing one. When creating the slice, existing values would be overwritten so this should only be used when it does not exist.

The following example shows both cases to consider while writing a patch for adding a value to a slice. This patch targets to add a file to the files slice of a KubeadmConfigTemplate which has omitempty set.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: my-clusterclass
spec:
  ...
  patches:
  - name: add file
    definitions:
    - selector:
        apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
        kind: KubeadmConfigTemplate
      jsonPatches:
      - op: add
        path: /spec/template/spec/files/-
        value:
          content: Some content.
          path: /some/file
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
metadata:
  name: "quick-start-default-worker-bootstraptemplate"
spec:
  template:
    spec:
      ...
      files:
      - content: Some other content
        path: /some/other/file

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: my-clusterclass
spec:
  ...
  patches:
  - name: add file
    definitions:
    - selector:
        apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
        kind: KubeadmConfigTemplate
      jsonPatches:
      - op: add
        path: /spec/template/spec/files
        value:
        - content: Some content.
          path: /some/file
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta2
kind: KubeadmConfigTemplate
metadata:
  name: "quick-start-default-worker-bootstraptemplate"
spec:
  template:
    spec:
      ...

Changing a ClusterClass

Selecting a strategy

When planning a change to a ClusterClass, users should always take into consideration how those changes might impact the existing Clusters already using the ClusterClass, if any.

There are two strategies for defining how a ClusterClass change rolls out to existing Clusters:

• Roll out ClusterClass changes to existing Cluster in a controlled/incremental fashion.
• Roll out ClusterClass changes to all the existing Cluster immediately.

The first strategy is the recommended choice for people starting with ClusterClass; it requires the users to create a new ClusterClass with the expected changes, and then rebase each Cluster to use the newly created ClusterClass.

By splitting the change to the ClusterClass and its rollout to Clusters into separate steps the user will reduce the risk of introducing unexpected changes on existing Clusters, or at least limit the blast radius of those changes to a small number of Clusters already rebased (in fact it is similar to a canary deployment).

The second strategy listed above instead requires changing a ClusterClass “in place”, which can be simpler and faster than creating a new ClusterClass. However, this approach means that changes are immediately propagated to all the Clusters already using the modified ClusterClass. Any operation involving many Clusters at the same time has intrinsic risks, and it can impact heavily on the underlying infrastructure in case the operation triggers machine rollout across the entire fleet of Clusters.

However, regardless of which strategy you are choosing to implement your changes to a ClusterClass, please make sure to:

• Plan ClusterClass changes before applying them.
• Understand what Compatibility Checks are and how to prevent changes that can lead to non-functional Clusters.

If instead you are interested in understanding more about which kind of
effects you should expect on the Clusters, or if you are interested in additional details about the internals of the topology reconciler you can start reading the notes in the Plan ClusterClass changes documentation or looking at the reference documentation at the end of this page.

Changing ClusterClass templates

Templates are an integral part of a ClusterClass, and thus the same considerations described in the previous paragraph apply. When changing a template referenced in a ClusterClass users should also always plan for how the change should be propagated to the existing Clusters and choose the strategy that best suits expectations.

According to the Cluster API operational practices, the recommended way for updating templates is by template rotation:

• Create a new template
• Update the template reference in the ClusterClass
• Delete the old template

Also in case of changes to the ClusterClass templates, please make sure to:

• Plan ClusterClass changes before applying them.
• Understand what Compatibility Checks are and how to prevent changes that can lead to non-functional Clusters.

You can learn more about this reading the notes in the Plan ClusterClass changes documentation or looking at the reference documentation at the end of this page.

Rebase

Rebasing is an operational practice for transitioning a Cluster from one ClusterClass to another, and the operation can be triggered by simply changing the value in Cluster.spec.topology.class.

Also in this case, please make sure to:

• Plan ClusterClass changes before applying them.
• Understand what Compatibility Checks are and how to prevent changes that can lead to non-functional Clusters.

You can learn more about this reading the notes in the Plan ClusterClass changes documentation or looking at the reference documentation at the end of this page.

Compatibility Checks

When changing a ClusterClass, the system validates the required changes according to a set of compatibility rules to prevent changes which would lead to a non-functional Cluster, e.g. changing the InfrastructureProvider from AWS to Azure.

If the proposed changes are evaluated as dangerous, the operation is rejected.

Planning ClusterClass changes

Some general notes that can help you to understand what you should expect when planning your ClusterClass changes:

• Users should expect the resources in a Cluster (e.g. MachineDeployments) to behave consistently no matter if a change is applied via a ClusterClass or directly as you do in a Cluster without a ClusterClass. In other words, if someone changes something on a KCP object triggering a control plane Machines rollout, you should expect the same to happen when the same change is applied to the KCP template in ClusterClass.

• User should expect the Cluster topology to change consistently irrespective of how the change has been implemented inside the ClusterClass or applied to the ClusterClass. In other words, if you change a template field “in place”, or if you rotate the template referenced in the ClusterClass by pointing to a new template with the same field changed, or if you change the same field via a patch, the effects on the Cluster are the same.

See reference for more details.

Reference

Effects on the Clusters

The following table documents the effects each ClusterClass change can have on a Cluster; Similar considerations apply to changes introduced by changes in Cluster.Topology or by changes introduced by patches.

NOTE: for people used to operating Cluster API without Cluster Class, it could also help to keep in mind that the underlying objects like control plane and MachineDeployment act in the same way with and without a ClusterClass.

How the topology controller reconciles template fields

The topology reconciler enforces values defined in the ClusterClass templates into the topology owned objects in a Cluster.

More specifically, the topology controller uses Server Side Apply to write/patch topology owned objects; using SSA allows other controllers to co-author the generated objects, like e.g. adding info for subnets in CAPA.

A corollary of the behaviour described above is that it is technically possible to change fields in the object which are not derived from the templates and patches, but we advise against using the possibility or making ad-hoc changes in generated objects unless otherwise needed for a workaround. It is always preferable to improve ClusterClasses by supporting new Cluster variants in a reusable way.

Operating a managed Cluster

The spec.topology field added to the Cluster object as part of ClusterClass allows changes made on the Cluster to be propagated across all relevant objects. This means the Cluster object can be used as a single point of control for making changes to objects that are part of the Cluster, including the ControlPlane and MachineDeployments.

A managed Cluster can be used to:

• Upgrade a Cluster
• Scale a ControlPlane
• Scale a MachineDeployment
• Add a MachineDeployment
• Use variables in a Cluster
• Rebase a Cluster to a different ClusterClass
• Upgrading Cluster API
• Tips and tricks

Upgrade a Cluster

Using a managed topology the operation to upgrade a Kubernetes cluster is a one-touch operation. Let’s assume we have created a CAPD cluster with ClusterClass and specified Kubernetes v1.21.2 (as documented in the Quick Start guide). Specifying the version is done when running clusterctl generate cluster. Looking at the cluster, the version of the control plane and the MachineDeployments is v1.21.2.

> kubectl get kubeadmcontrolplane,machinedeployments

NAME                                                                              CLUSTER                   INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE     VERSION
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/clusterclass-quickstart-XXXX    clusterclass-quickstart   true          true                   1          1       1         0             2m21s   v1.21.2

NAME                                                                             CLUSTER                   REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE     VERSION
machinedeployment.cluster.x-k8s.io/clusterclass-quickstart-linux-workers-XXXX    clusterclass-quickstart   1          1       1         0             Running   2m21s   v1.21.2

To update the Cluster the only change needed is to the version field under spec.topology in the Cluster object.

Change 1.21.2 to 1.22.0 as below.

kubectl patch cluster clusterclass-quickstart --type json --patch '[{"op": "replace", "path": "/spec/topology/version", "value": "v1.22.0"}]'

The patch will make the following change to the Cluster yaml:

   spec:
     topology:
      classRef:
        name: quick-start
+     version: v1.22.0
-     version: v1.21.2 

Important Note: A +2 minor Kubernetes version upgrade is not allowed in Cluster Topologies. This is to align with existing control plane providers, like KubeadmControlPlane provider, that limit a +2 minor version upgrade. Example: Upgrading from 1.21.2 to 1.23.0 is not allowed.

The upgrade will take some time to roll out as it will take place machine by machine with older versions of the machines only being removed after healthy newer versions come online.

To watch the update progress run:

watch kubectl get kubeadmcontrolplane,machinedeployments

After a few minutes the upgrade will be complete and the output will be similar to:

NAME                                                                              CLUSTER                   INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE     VERSION
kubeadmcontrolplane.controlplane.cluster.x-k8s.io/clusterclass-quickstart-XXXX    clusterclass-quickstart   true          true                   1          1       1         0             7m29s   v1.22.0

NAME                                                                             CLUSTER                   REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE     VERSION
machinedeployment.cluster.x-k8s.io/clusterclass-quickstart-linux-workers-XXXX    clusterclass-quickstart   1          1       1         0             Running   7m29s   v1.22.0

Scale a MachineDeployment

When using a managed topology scaling of MachineDeployments, both up and down, should be done through the Cluster topology.

Assume we have created a CAPD cluster with ClusterClass and Kubernetes v1.23.3 (as documented in the Quick Start guide). Initially we should have a MachineDeployment with 3 replicas. Running

kubectl get machinedeployments

Will give us:

NAME                                                            CLUSTER           REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE   VERSION
machinedeployment.cluster.x-k8s.io/capi-quickstart-md-0-XXXX   capi-quickstart   3          3       3         0             Running   21m   v1.23.3

We can scale up or down this MachineDeployment through the Cluster object by changing the replicas field under /spec/topology/workers/machineDeployments/0/replicas The 0 in the path refers to the position of the target MachineDeployment in the list of our Cluster topology. As we only have one MachineDeployment we’re targeting the first item in the list under /spec/topology/workers/machineDeployments/.

To change this value with a patch:

kubectl patch cluster capi-quickstart --type json --patch '[{"op": "replace", "path": "/spec/topology/workers/machineDeployments/0/replicas",  "value": 1}]'

This patch will make the following changes on the Cluster yaml:

   spec:
     topology:
       workers:
         machineDeployments:
         - class: default-worker
           name: md-0
           metadata: {}
+          replicas: 1
-          replicas: 3

After a minute the MachineDeployment will have scaled down to 1 replica:

NAME                         CLUSTER           REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE   VERSION
capi-quickstart-md-0-XXXXX  capi-quickstart   1          1       1         0             Running   25m   v1.23.3

As well as scaling a MachineDeployment, Cluster operators can edit the labels and annotations applied to a running MachineDeployment using the Cluster topology as a single point of control.

Add a MachineDeployment

MachineDeployments in a managed Cluster are defined in the Cluster’s topology. Cluster operators can add a MachineDeployment to a living Cluster by adding it to the cluster.spec.topology.workers.machineDeployments field.

Assume we have created a CAPD cluster with ClusterClass and Kubernetes v1.23.3 (as documented in the Quick Start guide). Initially we should have a single MachineDeployment with 3 replicas. Running

kubectl get machinedeployments

Will give us:

NAME                                                            CLUSTER           REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE   VERSION
machinedeployment.cluster.x-k8s.io/capi-quickstart-md-0-XXXX   capi-quickstart   3          3       3         0             Running   21m   v1.23.3

A new MachineDeployment can be added to the Cluster by adding a new MachineDeployment spec under /spec/topology/workers/machineDeployments/. To do so we can patch our Cluster with:

kubectl patch cluster capi-quickstart --type json --patch '[{"op": "add", "path": "/spec/topology/workers/machineDeployments/-",  "value": {"name": "second-deployment", "replicas": 1, "class": "default-worker"} }]'

This patch will make the below changes on the Cluster yaml:

   spec:
     topology:
       workers:
         machineDeployments:
         - class: default-worker
           metadata: {}
           replicas: 3
           name: md-0
+        - class: default-worker
+          metadata: {}
+          replicas: 1
+          name: second-deployment

After a minute to scale the new MachineDeployment we get:

NAME                                      CLUSTER           REPLICAS   READY   UPDATED   UNAVAILABLE   PHASE     AGE   VERSION
capi-quickstart-md-0-XXXX                 capi-quickstart   1          1       1         0             Running   39m   v1.23.3
capi-quickstart-second-deployment-XXXX    capi-quickstart   1          1       1         0             Running   99s   v1.23.3

Our second deployment uses the same underlying MachineDeployment class default-worker as our initial deployment. In this case they will both have exactly the same underlying machine templates. In order to modify the templates MachineDeployments are based on take a look at Changing a ClusterClass.

A similar process as that described here - removing the MachineDeployment from cluster.spec.topology.workers.machineDeployments - can be used to delete a running MachineDeployment from an active Cluster.

Scale a ControlPlane

When using a managed topology scaling of ControlPlane Machines, where the Cluster is using a topology that includes ControlPlane MachineInfrastructure, should be done through the Cluster topology.

This is done by changing the ControlPlane replicas field at /spec/topology/controlPlane/replica in the Cluster object. The command is:

kubectl patch cluster capi-quickstart --type json --patch '[{"op": "replace", "path": "/spec/topology/controlPlane/replicas",  "value": 1}]'

This patch will make the below changes on the Cluster yaml:

   spec:
      topology:
        controlPlane:
          metadata: {}
+         replicas: 1
-         replicas: 3

As well as scaling a ControlPlane, Cluster operators can edit the labels and annotations applied to a running ControlPlane using the Cluster topology as a single point of control.

Use variables

A ClusterClass can use variables and patches in order to allow flexible customization of Clusters derived from a ClusterClass. Variable definition allows two or more Cluster topologies derived from the same ClusterClass to have different specs, with the differences controlled by variables in the Cluster topology.

Assume we have created a CAPD cluster with ClusterClass and Kubernetes v1.23.3 (as documented in the Quick Start guide). Our Cluster has a variable etcdImageTag as defined in the ClusterClass. The variable is not set on our Cluster. Some variables, depending on their definition in a ClusterClass, may need to be specified by the Cluster operator for every Cluster created using a given ClusterClass.

In order to specify the value of a variable all we have to do is set the value in the Cluster topology.

We can see the current unset variable with:

kubectl get cluster capi-quickstart -o jsonpath='{.spec.topology.variables[1]}'                                     

Which will return something like:

{"name":"etcdImageTag","value":""}

In order to run a different version of etcd in new ControlPlane machines - the part of the spec this variable sets - change the value using the below patch:

kubectl patch cluster capi-quickstart --type json --patch '[{"op": "replace", "path": "/spec/topology/variables/1/value",  "value": "3.5.0"}]'

Running the patch makes the following change to the Cluster yaml:

   spec:
     topology:
       variables:
       - name: imageRepository
         value: registry.k8s.io
       - name: etcdImageTag
         value: ""
       - name: coreDNSImageTag
+        value: "3.5.0"
-        value: ""

Retrieving the variable value from the Cluster object, with kubectl get cluster capi-quickstart -o jsonpath='{.spec.topology.variables[1]}' we can see:

{"name":"etcdImageTag","value":"3.5.0"}

Note: Changing the etcd version may have unintended impacts on a running Cluster. For safety the cluster should be reapplied after running the above variable patch.

Rebase a Cluster

To perform more significant changes using a Cluster as a single point of control, it may be necessary to change the ClusterClass that the Cluster is based on. This is done by changing the class referenced in /spec/topology/class.

To read more about changing an underlying class please refer to ClusterClass rebase.

Tips and tricks

Users should always aim at ensuring the stability of the Cluster and of the applications hosted on it while using spec.topology as a single point of control for making changes to the objects that are part of the Cluster.

Following recommendation apply:

• If possible, avoid concurrent changes to control-plane and/or MachineDeployments to prevent excessive turnover on the underlying infrastructure or bottlenecks in the Cluster trying to move workloads from one machine to the other.
• Keep machine labels and annotation stable, because changing those values requires machines rollouts; also, please note that machine labels and annotation are not propagated to Kubernetes nodes; see metadata propagation.
• While upgrading a Cluster, if possible avoid any other concurrent change to the Cluster; please note that you can rely on version-aware patches to ensure the Cluster adapts to the new Kubernetes version in sync with the upgrade workflow.

For more details about how changes can affect a Cluster, please look at reference.

Upgrading Cluster API

There are some special considerations for ClusterClass regarding Cluster API upgrades when the upgrade includes a bump of the apiVersion of infrastructure, bootstrap or control plane provider CRDs.

The recommended approach is to first upgrade Cluster API and then update the apiVersions in the ClusterClass references afterwards. By following above steps, there won’t be any disruptions of the reconciliation as the Cluster topology controller is able to reconcile the Cluster even with the old apiVersions in the ClusterClass.

Note: The apiVersions in ClusterClass cannot be updated before Cluster API because the new apiVersions don’t exist in the management cluster before the Cluster API upgrade.

In general the Cluster topology controller always uses exactly the versions of the CRDs referenced in the ClusterClass. This means in the following example the Cluster topology controller will always use v1beta1 when reconciling/applying patches for the infrastructure ref, even if the DockerClusterTemplate already has a v1beta2 apiVersion.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
metadata:
  name: quick-start
  namespace: default
spec:
  infrastructure:
    templateRef:
      apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
      kind: DockerClusterTemplate
...

Experimental Feature: Runtime SDK (alpha)

The Runtime SDK feature provides an extensibility mechanism that allows systems, products, and services built on top of Cluster API to hook into a workload cluster’s lifecycle.

Feature gate name: RuntimeSDK

Variable name to enable/disable the feature gate: EXP_RUNTIME_SDK

Additional documentation:

• Background information:
  • Runtime SDK CAEP
  • Topology Mutation Hook CAEP
  • Runtime Hooks for Add-on Management CAEP
• For Runtime Extension developers:
  • Implementing Runtime Extensions
  • Implementing In-Place Update Hooks Extensions
  • Implementing Lifecycle Hook Extensions
  • Implementing Topology Mutation Hook Extensions
  • Implementing Upgrade Plan Runtime Extensions
• For Cluster operators:
  • Deploying Runtime Extensions

Implementing Runtime Extensions

Introduction

As a developer building systems on top of Cluster API, if you want to hook into the Cluster’s lifecycle via a Runtime Hook, you have to implement a Runtime Extension handling requests according to the OpenAPI specification for the Runtime Hook you are interested in.

Runtime Extensions by design are very powerful and flexible, however given that with great power comes great responsibility, a few key consideration should always be kept in mind (more details in the following sections):

• Runtime Extensions are components that should be designed, written and deployed with great caution given that they can affect the proper functioning of the Cluster API runtime.
• Cluster administrators should carefully vet any Runtime Extension registration, thus preventing malicious components from being added to the system.

Please note that following similar practices is already commonly accepted in the Kubernetes ecosystem for Kubernetes API server admission webhooks. Runtime Extensions share the same foundation and most of the same considerations/concerns apply.

Implementation

As mentioned above as a developer building systems on top of Cluster API, if you want to hook in the Cluster’s lifecycle via a Runtime Extension, you have to implement an HTTPS server handling a discovery request and a set of additional requests according to the OpenAPI specification for the Runtime Hook you are interested in.

The following shows a minimal example of a Runtime Extension server implementation:

package main

import (
	"context"
	"flag"
	"net/http"
	"os"

	"github.com/spf13/pflag"
	cliflag "k8s.io/component-base/cli/flag"
	"k8s.io/component-base/logs"
	logsv1 "k8s.io/component-base/logs/api/v1"
	"k8s.io/klog/v2"
	ctrl "sigs.k8s.io/controller-runtime"

	runtimehooksv1 "sigs.k8s.io/cluster-api/api/runtime/hooks/v1alpha1"
	runtimecatalog "sigs.k8s.io/cluster-api/exp/runtime/catalog"
	"sigs.k8s.io/cluster-api/exp/runtime/server"
)

var (
	// catalog contains all information about RuntimeHooks.
	catalog = runtimecatalog.New()

	// Flags.
	profilerAddress string
	webhookPort     int
	webhookCertDir  string
	logOptions      = logs.NewOptions()
)

func init() {
	// Adds to the catalog all the RuntimeHooks defined in cluster API.
	_ = runtimehooksv1.AddToCatalog(catalog)
}

// InitFlags initializes the flags.
func InitFlags(fs *pflag.FlagSet) {
	// Initialize logs flags using Kubernetes component-base machinery.
	logsv1.AddFlags(logOptions, fs)

	// Add test-extension specific flags
	fs.StringVar(&profilerAddress, "profiler-address", "",
		"Bind address to expose the pprof profiler (e.g. localhost:6060)")

	fs.IntVar(&webhookPort, "webhook-port", 9443,
		"Webhook Server port")

	fs.StringVar(&webhookCertDir, "webhook-cert-dir", "/tmp/k8s-webhook-server/serving-certs/",
		"Webhook cert dir.")
}

func main() {
	// Creates a logger to be used during the main func.
	setupLog := ctrl.Log.WithName("setup")

	// Initialize and parse command line flags.
	InitFlags(pflag.CommandLine)
	pflag.CommandLine.SetNormalizeFunc(cliflag.WordSepNormalizeFunc)
	pflag.CommandLine.AddGoFlagSet(flag.CommandLine)
	// Set log level 2 as default.
	if err := pflag.CommandLine.Set("v", "2"); err != nil {
		setupLog.Error(err, "Failed to set default log level")
		os.Exit(1)
	}
	pflag.Parse()

	// Validates logs flags using Kubernetes component-base machinery and applies them
	if err := logsv1.ValidateAndApply(logOptions, nil); err != nil {
		setupLog.Error(err, "Unable to start extension")
		os.Exit(1)
	}

	pflag.CommandLine.VisitAll(func(flag *pflag.Flag) {
		klog.V(1).Infof("FLAG: --%s=%q", flag.Name, flag.Value)
	})

	// Add the klog logger in the context.
	ctrl.SetLogger(klog.Background())

	// Initialize the golang profiler server, if required.
	if profilerAddress != "" {
		klog.Infof("Profiler listening for requests at %s", profilerAddress)
		go func() {
			klog.Info(http.ListenAndServe(profilerAddress, nil))
		}()
	}

	// Create a http server for serving runtime extensions
	webhookServer, err := server.New(server.Options{
		Catalog: catalog,
		Port:    webhookPort,
		CertDir: webhookCertDir,
	})
	if err != nil {
		setupLog.Error(err, "Error creating webhook server")
		os.Exit(1)
	}

	// Register extension handlers.
	if err := webhookServer.AddExtensionHandler(server.ExtensionHandler{
		Hook:        runtimehooksv1.BeforeClusterCreate,
		Name:        "before-cluster-create",
		HandlerFunc: DoBeforeClusterCreate,
	}); err != nil {
		setupLog.Error(err, "Error adding handler")
		os.Exit(1)
	}
	if err := webhookServer.AddExtensionHandler(server.ExtensionHandler{
		Hook:        runtimehooksv1.BeforeClusterUpgrade,
		Name:        "before-cluster-upgrade",
		HandlerFunc: DoBeforeClusterUpgrade,
	}); err != nil {
		setupLog.Error(err, "Error adding handler")
		os.Exit(1)
	}

	// Setup a context listening for SIGINT.
	ctx := ctrl.SetupSignalHandler()

	// Start the https server.
	setupLog.Info("Starting Runtime Extension server")
	if err := webhookServer.Start(ctx); err != nil {
		setupLog.Error(err, "Error running webhook server")
		os.Exit(1)
	}
}

func DoBeforeClusterCreate(ctx context.Context, request *runtimehooksv1.BeforeClusterCreateRequest, response *runtimehooksv1.BeforeClusterCreateResponse) {
	log := ctrl.LoggerFrom(ctx)
	log.Info("BeforeClusterCreate is called")
	// Your implementation
}

func DoBeforeClusterUpgrade(ctx context.Context, request *runtimehooksv1.BeforeClusterUpgradeRequest, response *runtimehooksv1.BeforeClusterUpgradeResponse) {
	log := ctrl.LoggerFrom(ctx)
	log.Info("BeforeClusterUpgrade is called")
	// Your implementation
}

For a full example see our test extension.

Please note that a Runtime Extension server can serve multiple Runtime Hooks (in the example above BeforeClusterCreate and BeforeClusterUpgrade) at the same time. Each of them are handled at a different path, like the Kubernetes API server does for different API resources. The exact format of those paths is handled by the server automatically in accordance to the OpenAPI specification of the Runtime Hooks.

There is an additional Discovery endpoint which is automatically served by the Server. The Discovery endpoint returns a list of extension handlers to inform Cluster API which Runtime Hooks are implemented by this Runtime Extension server.

Please note that Cluster API is only able to enforce the correct request and response types as defined by a Runtime Hook version. Developers are fully responsible for all other elements of the design of a Runtime Extension implementation, including:

• To choose which programming language to use; please note that Golang is the language of choice, and we are not planning to test or provide tooling and libraries for other languages. Nevertheless, given that we rely on Open API and plain HTTPS calls, other languages should just work but support will be provided at best effort.
• To choose if a dedicated or a shared HTTPS Server is used for the Runtime Extension (it can be e.g. also used to serve a metric endpoint).

When using Golang the Runtime Extension developer can benefit from the following packages (provided by the sigs.k8s.io/cluster-api module) as shown in the example above:

• api/runtime/hooks/v1alpha1 contains the Runtime Hook Golang API types, which are also used to generate the OpenAPI specification.
• exp/runtime/catalog provides the Catalog object to register Runtime Hook definitions. The Catalog is then used by the server package to handle requests. Catalog is similar to the runtime.Scheme of the k8s.io/apimachinery/pkg/runtime package, but it is designed to store Runtime Hook registrations.
• exp/runtime/server provides a Server object which makes it easy to implement a Runtime Extension server. The Server will automatically handle tasks like Marshalling/Unmarshalling requests and responses. A Runtime Extension developer only has to implement a strongly typed function that contains the actual logic.

Guidelines

While writing a Runtime Extension the following important guidelines must be considered:

Timeouts

Runtime Extension processing adds to reconcile durations of Cluster API controllers. They should respond to requests as quickly as possible, typically in milliseconds. Runtime Extension developers can decide how long the Cluster API Runtime should wait for a Runtime Extension to respond before treating the call as a failure (max is 30s) by returning the timeout during discovery. Of course a Runtime Extension can trigger long-running tasks in the background, but they shouldn’t block synchronously.

Availability

Runtime Extension failure could result in errors in handling the workload clusters lifecycle, and so the implementation should be robust, have proper error handling, avoid panics, etc. Failure policies can be set up to mitigate the negative impact of a Runtime Extension on the Cluster API Runtime, but this option can’t be used in all cases (see Error Management).

Blocking Hooks

A Runtime Hook can be defined as “blocking” - e.g. the BeforeClusterUpgrade hook allows a Runtime Extension to prevent the upgrade from starting. A Runtime Extension registered for the BeforeClusterUpgrade hook can block by returning a non-zero retryAfterSeconds value. Following consideration apply:

• The system might decide to retry the same Runtime Extension even before the retryAfterSeconds period expires, e.g. due to other changes in the Cluster, so retryAfterSeconds should be considered as an approximate maximum time before the next reconcile.
• If there is more than one Runtime Extension registered for the same Runtime Hook and more than one returns retryAfterSeconds, the shortest non-zero value will be used.
• If there is more than one Runtime Extension registered for the same Runtime Hook and at least one returns retryAfterSeconds, all Runtime Extensions will be called again.

Detailed description of what “blocking” means for each specific Runtime Hooks is documented case by case in the hook-specific implementation documentation (e.g. Implementing Lifecycle Hook Runtime Extensions).

Side Effects

It is recommended that Runtime Extensions should avoid side effects if possible, which means they should operate only on the content of the request sent to them, and not make out-of-band changes. If side effects are required, rules defined in the following sections apply.

Idempotence

An idempotent Runtime Extension is able to succeed even in case it has already been completed before (the Runtime Extension checks current state and changes it only if necessary). This is necessary because a Runtime Extension may be called many times after it already succeeded because other Runtime Extensions for the same hook may not succeed in the same reconcile.

A practical example that explains why idempotence is relevant is the fact that extensions could be called more than once for the same lifecycle transition, e.g.

• Two Runtime Extensions are registered for the BeforeClusterUpgrade hook.
• Before a Cluster upgrade is started both extensions are called, but one of them temporarily blocks the operation by asking to retry after 30 seconds.
• After 30 seconds the system retries the lifecycle transition, and both extensions are called again to re-evaluate if it is now possible to proceed with the Cluster upgrade.

Avoid dependencies

Each Runtime Extension should accomplish its task without depending on other Runtime Extensions. Introducing dependencies across Runtime Extensions makes the system fragile, and it is probably a consequence of poor “Separation of Concerns” between extensions.

Deterministic result

A deterministic Runtime Extension is implemented in such a way that given the same input it will always return the same output.

Some Runtime Hooks, e.g. like external patches, might explicitly request for corresponding Runtime Extensions to support this property. But we encourage developers to follow this pattern more generally given that it fits well with practices like unit testing and generally makes the entire system more predictable and easier to troubleshoot.

Error messages

RuntimeExtension authors should be aware that error messages might be surfaced as conditions in Kubernetes resources and recorded in Cluster API controller’s logs. As a consequence:

• Error message must not contain any sensitive information.
• Error message must be deterministic, and must avoid to including timestamps or values changing at every call.
• Error message must not contain external errors when it’s not clear if those errors are deterministic (e.g. errors return from cloud APIs).

ExtensionConfig

To register your runtime extension apply the ExtensionConfig resource in the management cluster, including your CA certs, ClusterIP service associated with the app and namespace, and the target namespace for the given extension. Once created, the extension will detect the associated service and discover the associated Hooks. For clarification, you can check the status of the ExtensionConfig. Below is an example of ExtensionConfig -

apiVersion: runtime.cluster.x-k8s.io/v1beta2
kind: ExtensionConfig
metadata:
  annotations:
    runtime.cluster.x-k8s.io/inject-ca-from-secret: default/test-runtime-sdk-svc-cert
  name: test-runtime-sdk-extensionconfig
spec:
  clientConfig:
    service:
      name: test-runtime-sdk-svc
      namespace: default # Note: this assumes the test extension get deployed in the default namespace
      port: 443
  namespaceSelector:
    matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: In
        values:
          - default # Note: this assumes the test extension is used by Cluster in the default namespace only

Settings

Settings can be added to the ExtensionConfig object in the form of a map with string keys and values. These settings are sent with each request to hooks registered by that ExtensionConfig. Extension developers can implement behavior in their extensions to alter behavior based on these settings. Settings should be well documented by extension developers so that ClusterClass authors can understand usage and expected behaviour.

Settings can be provided for individual external patches by providing them in the ClusterClass .spec.patches[*].external.settings. This can be used to overwrite settings at the ExtensionConfig level for that patch.

Error management

In case a Runtime Extension returns an error, the error will be handled according to the corresponding failure policy defined in the response of the Discovery call.

If the failure policy is Ignore the error is going to be recorded in the controller’s logs, but the processing will continue. However we recognize that this failure policy cannot be used in most of the use cases because Runtime Extension implementers want to ensure that the task implemented by an extension is completed before continuing with the cluster’s lifecycle.

If instead the failure policy is Fail the system will retry the operation until it passes. The following general considerations apply:

• It is the responsibility of Cluster API components to surface Runtime Extension errors using conditions.
• Operations will be retried with an exponential backoff or whenever the state of a Cluster changes (we are going to rely on controller runtime exponential backoff/watches).
• If there is more than one Runtime Extension registered for the same Runtime Hook and at least one of them fails, all the registered Runtime Extension will be retried. See Idempotence

Additional considerations about errors that apply only to a specific Runtime Hook will be documented in the hook-specific implementation documentation.

Tips & tricks

Make sure to add the ExtensionConfig object to the YAML manifest used to deploy the runtime extensions (see Extensionsconfig for more details).

After you implemented and deployed a Runtime Extension you can manually test it by sending HTTP requests. This can be for example done via kubectl:

Via kubectl create --raw:

# Send a Discovery Request to the webhook-service in namespace default with protocol https on port 443:
kubectl create --raw '/api/v1/namespaces/default/services/https:webhook-service:443/proxy/hooks.runtime.cluster.x-k8s.io/v1alpha1/discovery' \
  -f <(echo '{"apiVersion":"hooks.runtime.cluster.x-k8s.io/v1alpha1","kind":"DiscoveryRequest"}') | jq

Via kubectl proxy and curl:

# Open a proxy with kubectl and then use curl to send the request
## First terminal:
kubectl proxy
## Second terminal:
curl -X 'POST' 'http://127.0.0.1:8001/api/v1/namespaces/default/services/https:webhook-service:443/proxy/hooks.runtime.cluster.x-k8s.io/v1alpha1/discovery' \
  -d '{"apiVersion":"hooks.runtime.cluster.x-k8s.io/v1alpha1","kind":"DiscoveryRequest"}' | jq

For more details about the API of the Runtime Extensions please see Swagger UI. For more details on proxy support please see Proxies in Kubernetes.

Implementing in-place update hooks

Introduction

The proposal for in-place updates in Cluster API introduced extensions allowing users to execute changes on existing machines without deleting the Machine and creating a new one.

Notably, the Cluster API user experience remains the same as of today no matter of the in-place update feature is enabled or not e.g. in order to trigger a MachineDeployment rollout, you have to rotate a template, etc.

Users should care ONLY about the desired state (as of today).

Cluster API is responsible to choose the best strategy to achieve desired state, and with the introduction of update extensions, Cluster API is expanding the set of tools that can be used to achieve the desired state.

If external update extensions can not cover the totality of the desired changes, CAPI will fall back to Cluster API’s default, immutable rollouts.

Cluster API will be also responsible to determine which Machine/MachineSet should be updated, as well as to handle rollout options like MaxSurge/MaxUnavailable. With this regard:

• Machines updating in-place are considered not available, because in-place updates are always considered as potentially disruptive.
  • For control plane machines, if maxSurge is 1, a new machine must be created first, then as soon as there is “buffer” for in-place, in-place update can proceed.
    • KCP will not use in-place in case it will detect that it can impact health of the control plane.
  • For workers machines, if maxUnavailable is 0, a new machine must be created first, then as soon as there is “buffer” for in-place, in-place update can proceed.
    • When in-place is possible, the system should try to in-place update as many machines as possible. In practice, this means that maxSurge might not be fully used (it is used only for scale up by one if maxUnavailable=0).
  • No in-place updates are performed for workers machines when using rollout strategy OnDelete.

• Implementing in-place update hooks
  • Introduction
  • Guidelines
  • Definitions
    • CanUpdateMachine
    • CanUpdateMachineSet
    • UpdateMachine

Guidelines

All guidelines defined in Implementing Runtime Extensions apply to the implementation of Runtime Extensions for in-place update hooks as well.

In summary, Runtime Extensions are components that should be designed, written and deployed with great caution given that they can affect the proper functioning of the Cluster API runtime. A poorly implemented Runtime Extension could potentially block updates.

Following recommendations are especially relevant:

• Timeouts
• Idempotence
• Deterministic result
• Error messages
• Error management
• Avoid dependencies

Definitions

For additional details about the OpenAPI spec of the upgrade plan hooks, please download the runtime-sdk-openapi.yaml file and then open it from the Swagger UI.

CanUpdateMachine

This hook is called by KCP when performing the “can update in-place” for a control plane machine.

Example request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: CanUpdateMachineRequest
settings: <Runtime Extension settings>
current:
  machine:
    apiVersion: cluster.x-k8s.io/v1beta2
    kind: Machine
    metadata:
      name: test-cluster
      namespace: test-ns
    spec:
      ...
  infrastructureMachine:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachine
    metadata:
      name: test-cluster
      namespace: test-ns
    spec:
      ...
  boostrapConfig:
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfig
    metadata:
      name: test-cluster
      namespace: test-ns
    spec:
      ...
desired:
  machine:
    ...
  infrastructureMachine:
    ...
  boostrapConfig:
    ...

Note:

• All the objects will have the latest API version known by Cluster API.
• Only spec is provided, status fields are not included
• In a future release, when registering more than one extension for the CanUpdateMachine will be supported, the current state will already include changes that can be handled in-place by other runtime extensions.

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: CanUpdateMachineResponse
status: Success # or Failure
message: "error message if status == Failure"
machinePatch:
  patchType: JSONPatch
  patch: <JSON-patch>
infrastructureMachinePatch:
  ...
boostrapConfigPatch:
  ...

Note:

• Extensions should return per-object patches to be applied on current objects to indicate which changes they can handle in-place.
• Only fields in Machine/InfraMachine/BootstrapConfig spec have to be covered by patches
• Patches must be in JSONPatch or JSONMergePatch format

CanUpdateMachineSet

This hook is called by the MachineDeployment controller when performing the “can update in-place” for all the Machines controlled by a MachineSet.

Example request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: CanUpdateMachineSetRequest
settings: <Runtime Extension settings>
current:
  machineSet:
    apiVersion: cluster.x-k8s.io/v1beta2
    kind: MachineSet
    metadata:
      name: test-cluster
      namespace: test-ns
    spec:
      ...
  infrastructureMachineTemplate:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachineTemplate
    metadata:
      name: test-cluster
      namespace: test-ns
    spec:
      ...
  boostrapConfigTemplate:
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfigTemplate
    metadata:
      name: test-cluster
      namespace: test-ns
    spec:
      ...
desired:
  machineSet:
    ...
  infrastructureMachineTemplate:
    ...
  boostrapConfigTemplate:
    ...

Note:

• All the objects will have the latest API version known by Cluster API.
• Only spec is provided, status fields are not included
• In a future release, when registering more than one extension for the CanUpdateMachineSet will be supported, the current state will already include changes that can be handled in-place by other runtime extensions.

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: CanUpdateMachineSetResponse
status: Success # or Failure
message: "error message if status == Failure"
machineSetPatch:
  patchType: JSONPatch
  patch: <JSON-patch>
infrastructureMachineTemplatePatch:
  ...
boostrapConfigTemplatePatch:
  ...

Note:

• Extensions should return per-object patches to be applied on current objects to indicate which changes they can handle in-place.
• Only fields in MachineSet/InfraMachineTemplate/BootstrapConfigTemplate spec.template.spec have to be covered by patches
• Patches must be in JSONPatch or JSONMergePatch format

UpdateMachine

This hook is called by the Machine controller when performing the in-place updates for a Machine.

Example request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: UpdateMachineRequest
settings: <Runtime Extension settings>
desired:
  machine:
    apiVersion: cluster.x-k8s.io/v1beta2
    kind: Machine
    metadata:
      name: test-cluster
      namespace: test-ns
    spec:
      ...
  infrastructureMachine:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: VSphereMachine
    metadata:
      name: test-cluster
      namespace: test-ns
    spec:
      ...
  boostrapConfig:
    apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
    kind: KubeadmConfig
    metadata:
      name: test-cluster
      namespace: test-ns
    spec:
      ...

Note:

• Only desired is provided (the external updater extension should know current state of the Machine).
• Only spec is provided, status fields are not included

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: UpdateMachineResponse
status: Success # or Failure
message: "error message if status == Failure"
retryAfterSeconds: 10

Note:

• The status of the update operation is determined by the CommonRetryResponse fields:
  • Status=Success + RetryAfterSeconds > 0: update is in progress
  • Status=Success + RetryAfterSeconds = 0: update completed successfully
  • Status=Failure: update failed

Implementing Lifecycle Hook Runtime Extensions

Introduction

The lifecycle hooks allow hooking into the Cluster lifecycle. The following diagram provides an overview:

Please see the corresponding CAEP as well as the proposal for Chained and efficient upgrades for additional background information.

• Implementing Lifecycle Hook Runtime Extensions
  • Introduction
  • Guidelines
  • Definitions
    • BeforeClusterCreate
    • AfterControlPlaneInitialized
    • BeforeClusterUpgrade
    • BeforeControlPlaneUpgrade
    • AfterControlPlaneUpgrade
    • BeforeWorkersUpgrade
    • AfterWorkersUpgrade
    • AfterClusterUpgrade
    • BeforeClusterDelete

Guidelines

All guidelines defined in Implementing Runtime Extensions apply to the implementation of Runtime Extensions for lifecycle hooks as well.

In summary, Runtime Extensions are components that should be designed, written and deployed with great caution given that they can affect the proper functioning of the Cluster API runtime. A poorly implemented Runtime Extension could potentially block lifecycle transitions from happening.

Following recommendations are especially relevant:

• Blocking and non-blocking
• Error messages
• Error management
• Avoid dependencies

Definitions

For additional details about the OpenAPI spec of the lifecycle hooks, please download the runtime-sdk-openapi.yaml file and then open it from the Swagger UI.

BeforeClusterCreate

This hook is called after the Cluster object has been created by the user, immediately before all the objects which are part of a Cluster topology(*) are going to be created. Runtime Extension implementers can use this hook to determine/prepare add-ons for the Cluster and block the creation of those objects until everything is ready.

Example Request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: BeforeClusterCreateRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
   name: test-cluster
   namespace: test-ns
  spec:
   ...
  status:
   ...

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: BeforeClusterCreateResponse
status: Success # or Failure
message: "error message if status == Failure"
retryAfterSeconds: 10

(*) The objects which are part of a Cluster topology are the infrastructure Cluster, the Control Plane, the MachineDeployments and the templates derived from the ClusterClass.

AfterControlPlaneInitialized

This hook is called after the Control Plane reports that the control plane is initialized, which means the API server can accept requests. This usually happens sometime during the first CP machine provisioning or immediately thereafter.

Runtime Extension implementers can use this hook to execute tasks, for example component installation on workload clusters, that are only possible once the Control Plane is available. This hook does not block any further changes to the Cluster.

Example Request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: AfterControlPlaneInitializedRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
   name: test-cluster
   namespace: test-ns
  spec:
   ...
  status:
   ...

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: AfterControlPlaneInitializedResponse
status: Success # or Failure
message: "error message if status == Failure"

BeforeClusterUpgrade

This hook is called after the Cluster object has been updated with a new spec.topology.version by the user, and immediately before the new version is going to be propagated to the control plane (*). Runtime Extension implementers can use this hook to execute pre-upgrade add-on tasks and block upgrades of the ControlPlane and Workers.

(*) Under normal circumstances spec.topology.version gets propagated to the control plane immediately; however if previous upgrades or worker machine rollouts are still in progress, the system waits for those operations to complete before starting the new upgrade.

Note: While the upgrade is blocked changes made to the Cluster Topology will be delayed propagating to the underlying objects while the object is waiting for upgrade. Example: modifying ControlPlane/MachineDeployments (think scale up), or creating new MachineDeployments will be delayed until the target ControlPlane/MachineDeployment is ready to pick up the upgrade. This ensures that the ControlPlane and MachineDeployments do not perform a rollout prematurely while waiting to be rolled out again for the version upgrade (no double rollouts). This also ensures that any version specific changes are only pushed to the underlying objects also at the correct version.

Example Request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: BeforeClusterUpgradeRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
    name: test-cluster
    namespace: test-ns
  spec:
    ...
  status:
    ...
fromKubernetesVersion: "v1.30.0"
toKubernetesVersion: "v1.33.0"
controlPlaneUpgrades:
  - version: v1.31.0
  - version: v1.32.3
  - version: v1.33.0
workersUpgrades:
  - version: v1.32.3
  - version: v1.33.0

Note: The controlPlaneUpgrades and the workersUpgrades fields contain the intermediate steps to reach the target version, which is also included in the list.

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: BeforeClusterUpgradeResponse
status: Success # or Failure
message: "error message if status == Failure"
retryAfterSeconds: 10

BeforeControlPlaneUpgrade

This hook is called before a new version is propagated to the control plane object, which happens as many times as defined by the upgrade plan.

Runtime Extension implementers can use this hook to execute pre-upgrade add-on tasks and block upgrades of the ControlPlane.

Note:

• When an upgrade is starting, BeforeControlPlaneUpgrade will be called after BeforeClusterUpgrade is completed.
• When an upgrade is in progress BeforeControlPlaneUpgrade will be called for each intermediate version that will be applied to the control plane (instead BeforeClusterUpgrade will be called only once at the beginning of the upgrade).

Example Request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: BeforeControlPlaneUpgradeRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
   name: test-cluster
   namespace: test-ns
  spec:
   ...
  status:
   ...
fromKubernetesVersion: "v1.30.0"
toKubernetesVersion: "v1.33.0"
controlPlaneUpgrades:
  - version: v1.31.0
  - version: v1.32.3
  - version: v1.33.0
workersUpgrades:
  - version: v1.32.3
  - version: v1.33.0

Note: The controlPlaneUpgrades and the workersUpgrades fields contain the intermediate steps to reach the target version, which is also included in the list.

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: BeforeControlPlaneUpgradeResponse
status: Success # or Failure
message: "error message if status == Failure"
retryAfterSeconds: 10

AfterControlPlaneUpgrade

This hook is called after the control plane has been upgraded to the version specified in spec.topology.version or to an intermediate version in the upgrade plan and:

• if workers upgrade can be skipped for this version and this is an intermediate version of an upgrade plan, immediately before calling the BeforeControlPlaneUpgrade hook for the next version in the upgrade plan.
• if workers upgrade must be performed for this version, immediately before calling the BeforeWorkersUpgrade hook for the same version.
• if the cluster does not have workers and this is the last version of an upgrade plan, immediately before calling the AfterClusterUpgrade hook.

Runtime Extension implementers can use this hook to execute post-upgrade add-on tasks and block upgrades to the next version of the control plane or to workers until everything is ready.

Note: While the MachineDeployments upgrade is blocked changes made to existing MachineDeployments and creating new MachineDeployments will be delayed while the object is waiting for upgrade. Example: modifying MachineDeployments (think scale up), or creating new MachineDeployments will be delayed until the target MachineDeployment is ready to pick up the upgrade. This ensures that the MachineDeployments do not perform a rollout prematurely while waiting to be rolled out again for the version upgrade (no double rollouts). This also ensures that any version specific changes are only pushed to the underlying objects also at the correct version.

Example Request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: AfterControlPlaneUpgradeRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
    name: test-cluster
    namespace: test-ns
  spec:
    ...
  status:
    ...
kubernetesVersion: "v1.30.0"
controlPlaneUpgrades:
  - version: v1.31.0
  - version: v1.32.3
  - version: v1.33.0
workersUpgrades:
  - version: v1.32.3
  - version: v1.33.0

Note: The controlPlaneUpgrades and the workersUpgrades fields contain the intermediate steps to reach the target version, which is also included in the list.

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: AfterControlPlaneUpgradeResponse
status: Success # or Failure
message: "error message if status == Failure"
retryAfterSeconds: 10

BeforeWorkersUpgrade

This hook is called before a new version is propagated to workers. Runtime Extension implementers can use this hook to execute pre-upgrade add-on tasks and block upgrades of Workers.

Note:

• This hook will be called only if workers upgrade must be performed for an intermediate version of a chained upgrade or when upgrading to the target spec.topology.version.

Example Request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: BeforeWorkersUpgradeRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
   name: test-cluster
   namespace: test-ns
  spec:
   ...
  status:
   ...
fromKubernetesVersion: "v1.30.0"
toKubernetesVersion: "v1.33.0"
controlPlaneUpgrades:
  - version: v1.31.0
  - version: v1.32.3
  - version: v1.33.0
workersUpgrades:
  - version: v1.32.3
  - version: v1.33.0

Note: The controlPlaneUpgrades and the workersUpgrades fields contain the intermediate steps to reach the target version, which is also included in the list.

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: BeforeWorkersUpgradeResponse
status: Success # or Failure
message: "error message if status == Failure"
retryAfterSeconds: 10

AfterWorkersUpgrade

This hook is called after all the workers have been upgraded to the version specified in spec.topology.version or to an intermediate version in the upgrade plan, and:

• if the upgrade plan is completed and the entire cluster is at spec.topology.version, immediately before calling the AfterClusterUpgrade hook.
• if the upgrade plan is not complete and the entire cluster is now at one of the intermediate versions, immediately before calling BeforeControlPlaneUpgrade hook for the next intermediate step; in this case, the hook will ensure the control plane can’t to move to the next version in the upgrade plan until AfterWorkersUpgrade is completed.

Example Request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: AfterWorkersUpgradeRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
   name: test-cluster
   namespace: test-ns
  spec:
   ...
  status:
   ...
kubernetesVersion: "v1.30.0"
controlPlaneUpgrades:
  - version: v1.31.0
  - version: v1.32.3
  - version: v1.33.0
workersUpgrades:
  - version: v1.32.3
  - version: v1.33.0

Note: The controlPlaneUpgrades and the workersUpgrades fields contain the intermediate steps to reach the target version, which is also included in the list.

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: AfterWorkersUpgradeResponse
status: Success # or Failure
message: "error message if status == Failure"
retryAfterSeconds: 10

AfterClusterUpgrade

This hook is called after the Cluster, control plane and workers have been upgraded to the version specified in spec.topology.version. Runtime Extensions implementers can use this hook to execute post-upgrade add-on tasks. This hook blocks new upgrades to start until it is completed.

Example Request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: AfterClusterUpgradeRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
   name: test-cluster
   namespace: test-ns
  spec:
   ...
  status:
   ...
kubernetesVersion: "v1.22.0"

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: AfterClusterUpgradeResponse
status: Success # or Failure
message: "error message if status == Failure"
retryAfterSeconds: 10

BeforeClusterDelete

This hook is called after the Cluster deletion has been triggered by the user and immediately before the topology of the Cluster is going to be deleted. Runtime Extension implementers can use this hook to execute cleanup tasks for the add-ons and block deletion of the Cluster and descendant objects until everything is ready.

Example Request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: BeforeClusterDeleteRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
   name: test-cluster
   namespace: test-ns
  spec:
   ...
  status:
   ...

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: BeforeClusterDeleteResponse
status: Success # or Failure
message: "error message if status == Failure"
retryAfterSeconds: 10

Implementing Topology Mutation Hook Runtime Extensions

Introduction

Three different hooks are called as part of Topology Mutation - two in the Cluster topology reconciler and one in the ClusterClass reconciler.

Cluster topology reconciliation

• GeneratePatches: GeneratePatches is responsible for generating patches for the entire Cluster topology.
• ValidateTopology: ValidateTopology is called after all patches have been applied and thus allow to validate the resulting objects.

ClusterClass reconciliation

• DiscoverVariables: DiscoverVariables is responsible for providing variable definitions for a specific external patch.

Please see the corresponding CAEP for additional background information.

• Implementing Topology Mutation Hook Runtime Extensions
  • Introduction
  • Guidelines
  • Definitions
  • Inline vs. external patches
  • External variable definitions
    • External variable discovery in the ClusterClass
    • Variable definition conflicts
    • Setting values for variables in the Cluster
  • Using one or multiple external patch extensions
  • Guidelines
    • Patch extension guidelines
    • Variable discovery guidelines
  • Definitions
    • GeneratePatches
    • ValidateTopology
    • DiscoverVariables
  • Dealing with Cluster API upgrades with apiVersion bumps

Guidelines

All guidelines defined in Implementing Runtime Extensions apply to the implementation of Runtime Extensions for topology mutation hooks as well.

In summary, Runtime Extensions are components that should be designed, written and deployed with great caution given that they can affect the proper functioning of the Cluster API runtime. A poorly implemented Runtime Extension could potentially block topology reconcile from happening.

Following recommendations are especially relevant:

• Idempotence
• Avoid side effects
• Deterministic result
• Error messages
• Error management
• Avoid dependencies

Definitions

For additional details about the OpenAPI spec of the topology mutation hooks, please download the runtime-sdk-openapi.yaml file and then open it from the Swagger UI.

Inline vs. external patches

Inline patches have the following advantages:

• Inline patches are easier when getting started with ClusterClass as they are built into the Cluster API core controller, no external component have to be developed and managed.

External patches have the following advantages:

• External patches can be individually written, unit tested and released/versioned.
• External patches can leverage the full feature set of a programming language and are thus not limited to the capabilities of JSON patches and Go templating.
• External patches can use external data (e.g. from cloud APIs) during patch generation.
• External patches can be easily reused across ClusterClasses.

External variable definitions

The DiscoverVariables hook can be used to supply variable definitions for use in external patches. These variable definitions are added to the status of any applicable ClusterClasses. Clusters using the ClusterClass can then set values for those variables.

External variable discovery in the ClusterClass

External variable definitions are discovered by calling the DiscoverVariables runtime hook. This hook is called from the ClusterClass reconciler. Once discovered the variable definitions are validated and stored in ClusterClass status.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
# metadata
spec:
    # Inline variable definitions
    variables:
    # This variable is unique and can be accessed globally.
    - name: no-proxy
      required: true
      schema:
        openAPIV3Schema:
          type: string
          default: "internal.com"
          example: "internal.com"
          description: "comma-separated list of machine or domain names excluded from using the proxy."
    # This variable is also defined by an external DiscoverVariables hook.
    - name: http-proxy
      schema:
        openAPIV3Schema:
          type: string
          default: "proxy.example.com"
          example: "proxy.example.com"
          description: "proxy for http calls."
    # External patch definitions.
    patches:
    - name: lbImageRepository
      external:
          generatePatchesExtension: generate-patches.k8s-upgrade-with-runtimesdk
          validateTopologyExtension: validate-topology.k8s-upgrade-with-runtimesdk
          ## Call variable discovery for this patch.
          discoverVariablesExtension: discover-variables.k8s-upgrade-with-runtimesdk
status:
    # observedGeneration is used to check that the current version of the ClusterClass is the same as that when the Status was previously written.
    # if metadata.generation isn't the same as observedGeneration Cluster using the ClusterClass should not reconcile.
    observedGeneration: xx
    # variables contains a list of all variable definitions, both inline and from external patches, that belong to the ClusterClass.
    variables:
      - name: no-proxy
        definitions:
          - from: inline
            required: true
            schema:
              openAPIV3Schema:
                type: string
                default: "internal.com"
                example: "internal.com"
                description: "comma-separated list of machine or domain names excluded from using the proxy."
      - name: http-proxy
        # definitionsConflict is true if there are non-equal definitions for a variable.
        # Note: This conflict has to be resolved, until then corresponding Clusters are not reconciled.
        definitionsConflict: true
        definitions:
          - from: inline
            schema:
              openAPIV3Schema:
                type: string
                default: "proxy.example.com"
                example: "proxy.example.com"
                description: "proxy for http calls."
          - from: lbImageRepository
            schema:
              openAPIV3Schema:
                type: string
                default: "different.example.com"
                example: "different.example.com"
                description: "proxy for http calls."

Variable definition conflicts

Variable definitions can be inline in the ClusterClass or from any number of external DiscoverVariables hooks. The source of a variable definition is recorded in the from field in ClusterClass .status.variables. Variables that are defined by an external DiscoverVariables hook will have the name of the patch they are associated with as the value of from. Variables that are defined in the ClusterClass .spec.variables will have inline as the value of from. Note: inline is a reserved name for patches. It cannot be used as the name of an external patch to avoid conflicts.

If all variables that share a name have equivalent schemas the variable definitions are not in conflict. The CAPI components will consider variable definitions to be equivalent when they share a name and their schema is exactly equal. If variables are in conflict the VariablesReconciled will be set to false and the conflict has to be resolved. While there are variable conflicts, corresponding Clusters will not be reconciled.

Note: We enforce that variable conflicts have to be resolved by ClusterClass authors, so that defining Cluster topology is as simply as possible for end users.

Setting values for variables in the Cluster

Variables that are defined with external variable definitions can be set like regular variables in Cluster .spec.topology.variables.

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
#metadata 
spec:
    topology:
      variables:
        - name: no-proxy
          value: "internal.domain.com"
        - name: http-proxy
          value: http://proxy.example2.com:1234

Using one or multiple external patch extensions

Some considerations:

• In general a single external patch extension is simpler than many, as only one extension then has to be built, deployed and managed.
• A single extension also requires less HTTP round-trips between the CAPI controller and the extension(s).
• With a single extension it is still possible to implement multiple logical features using different variables.
• When implementing multiple logical features in one extension it’s recommended that they can be conditionally enabled/disabled via variables (either via certain values or by their existence).
• Conway’s law might make it not feasible in large organizations to use a single extension. In those cases it’s important that boundaries between extensions are clearly defined.

Guidelines

For general Runtime Extension developer guidelines please refer to the guidelines in Implementing Runtime Extensions. This section outlines considerations specific to Topology Mutation hooks.

Patch extension guidelines

• Input validation: An External Patch Extension must always validate its input, i.e. it must validate that all variables exist, have the right type and it must validate the kind and apiVersion of the templates which should be patched.
• Timeouts: As External Patch Extensions are called during each Cluster topology reconciliation, they must respond as fast as possible (<=200ms) to avoid delaying individual reconciles and congestion.
• Availability: An External Patch Extension must be always available, otherwise Cluster topologies won’t be reconciled anymore.
• Side Effects: An External Patch Extension must not make out-of-band changes. If necessary external data can be retrieved, but be aware of performance impact.
• Deterministic results: For a given request (a set of templates and variables) an External Patch Extension must always return the same response (a set of patches). Otherwise the Cluster topology will never reach a stable state.
• Idempotence: An External Patch Extension must only return patches if changes to the templates are required, i.e. unnecessary patches when the template is already in the desired state must be avoided.
• Avoid Dependencies: An External Patch Extension must be independent of other External Patch Extensions. However if dependencies cannot be avoided, it is possible to control the order in which patches are executed via the ClusterClass.
• Error messages: For a given request (a set of templates and variables) an External Patch Extension must always return the same error message. Otherwise the system might become unstable due to controllers being overloaded by continuous changes to Kubernetes resources as these messages are reported as conditions. See error messages.

Variable discovery guidelines

• Distinctive variable names: Names should be carefully chosen, and if possible generic names should be avoided. Using a generic name could lead to conflicts if the variables defined for this patch are used in combination with other patches providing variables with the same name.
• Avoid breaking changes to variable definitions: Changing a variable definition can lead to problems on existing clusters because reconciliation will stop if variable values do not match the updated definition. When more than one variable with the same name is defined, changes to variable definitions can require explicit values for each patch. Updates to the variable definition should be carefully evaluated, and very well documented in extension release notes, so ClusterClass authors can evaluate impacts of changes before performing an upgrade.

Definitions

GeneratePatches

A GeneratePatches call generates patches for the entire Cluster topology. Accordingly the request contains all templates, the global variables and the template-specific variables. The response contains generated patches.

Example request:

• Generating patches for a Cluster topology is done via a single call to allow External Patch Extensions a holistic view of the entire Cluster topology. Additionally this allows us to reduce the number of round-trips.
• Each item in the request will contain the template as a raw object. Additionally information about where the template is used is provided via holderReference.

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: GeneratePatchesRequest
settings: <Runtime Extension settings>
variables:
- name: <variable-name>
  value: <variable-value>
  ...
items:
- uid: 7091de79-e26c-4af5-8be3-071bc4b102c9
  holderReference:
    apiVersion: cluster.x-k8s.io/v1beta1
    kind: MachineDeployment
    namespace: default
    name: cluster-md1-xyz
    fieldPath: spec.template.spec.infrastructureRef
  object:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AWSMachineTemplate
    spec:
    ...
  variables:
  - name: <variable-name>
    value: <variable-value>
    ...

Example Response:

• The response contains patches instead of full objects to reduce the payload.
• Templates in the request and patches in the response will be correlated via UIDs.
• Like inline patches, external patches are only allowed to change fields in metadata.{labels,annotations}, spec.template.spec and spec.template.metadata.{labels,annotations} (see inline patches documentation for additional details).
• Only fields below metadata.{labels,annotations}, spec.template.spec and spec.template.metadata can be patched. Note that patching metadata.{labels,annotations} makes only sense when another template will be originated from the template object linked in the ClusterClass (e.g. it instead makes sense patching VSphereMachineTemplate.metadata when this object is referenced from a MachineDeploymentClass because another VSphereMachineTemplate will be generated for each MachineDeployment using this class; instead it does not make sense patching KubeadmControlPlaneTemplate.metadata.labels because this field will be lost when generating the KubeadmControlPlane object for a Cluster).

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: GeneratePatchesResponse
status: Success # or Failure
message: "error message if status == Failure"
items:
- uid: 7091de79-e26c-4af5-8be3-071bc4b102c9
  patchType: JSONPatch
  patch: <JSON-patch>

We are considering to introduce a library to facilitate development of External Patch Extensions. It would provide capabilities like:

• Accessing builtin variables
• Extracting certain templates from a GeneratePatches request (e.g. all bootstrap templates)

If you are interested in contributing to this library please reach out to the maintainer team or feel free to open an issue describing your idea or use case.

ValidateTopology

A ValidateTopology call validates the topology after all patches have been applied. The request contains all templates of the Cluster topology, the global variables and the template-specific variables. The response contains the result of the validation.

Example Request:

• The request is the same as the GeneratePatches request except it doesn’t have uid fields. We don’t need them as we don’t have to correlate patches in the response.

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: ValidateTopologyRequest
settings: <Runtime Extension settings>
variables:
- name: <variable-name>
  value: <variable-value>
  ...
items:
- holderReference:
    apiVersion: cluster.x-k8s.io/v1beta1
    kind: MachineDeployment
    namespace: default
    name: cluster-md1-xyz
    fieldPath: spec.template.spec.infrastructureRef
  object:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AWSMachineTemplate
    spec:
    ...
  variables:
  - name: <variable-name>
    value: <variable-value>
    ...

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: ValidateTopologyResponse
status: Success # or Failure
message: "error message if status == Failure"

DiscoverVariables

A DiscoverVariables call returns definitions for one or more variables.

Example Request:

• The request is a simple call to the Runtime hook.

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: DiscoverVariablesRequest
settings: <Runtime Extension settings>

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: DiscoverVariablesResponse
status: Success # or Failure
message: ""
variables:
  - name: etcdImageTag 
    required: true
    schema:
      openAPIV3Schema:
        type: string
        default: "3.5.3-0" 
        example: "3.5.3-0"
        description: "etcdImageTag sets the tag for the etcd image."
  - name: preLoadImages
    required: false
    schema:
      openAPIV3Schema:
        default: []
        type: array
        items:
          type: string
        description: "preLoadImages sets the images for the Docker machines to preload."
  - name: podSecurityStandard
    required: false
    schema:
      openAPIV3Schema:
        type: object
        properties:
          enabled:
            type: boolean
            default: true
            description: "enabled enables the patches to enable Pod Security Standard via AdmissionConfiguration."
          enforce:
            type: string
            default: "baseline"
            description: "enforce sets the level for the enforce PodSecurityConfiguration mode. One of privileged, baseline, restricted."
          audit:
            type: string
            default: "restricted"
            description: "audit sets the level for the audit PodSecurityConfiguration mode. One of privileged, baseline, restricted."
          warn:
            type: string
            default: "restricted"
            description: "warn sets the level for the warn PodSecurityConfiguration mode. One of privileged, baseline, restricted."
...

Dealing with Cluster API upgrades with apiVersion bumps

There are some special considerations regarding Cluster API upgrades when the upgrade includes a bump of the apiVersion of infrastructure, bootstrap or control plane provider CRDs.

When calling external patches the Cluster topology controller is always sending the templates in the apiVersion of the references in the ClusterClass.

While inline patches are always referring to one specific apiVersion, external patch implementations are more flexible. They can be written in a way that they are able to handle multiple apiVersions of a CRD. This can be done by calculating patches differently depending on which apiVersion is received by the external patch implementation.

This allows users more flexibility during Cluster API upgrades:

Variant 1: External patch implementation supporting two apiVersions at the same time

1. Update Cluster API
2. Update the external patch implementation to be able to handle custom resources with the old and the new apiVersion
3. Update the references in ClusterClasses to use the new apiVersion

Note In this variant it doesn’t matter if Cluster API or the external patch implementation is updated first.

Variant 2: Deploy an additional instance of the external patch implementation which can handle the new apiVersion

1. Upgrade Cluster API
2. Deploy the new external patch implementation which is able to handle the new apiVersion
3. Update ClusterClasses to use the new apiVersion and the new external patch implementation
4. Remove the old external patch implementation as it’s not used anymore

Note In this variant it doesn’t matter if Cluster API is updated or the new external patch implementation is deployed first.

Implementing Upgrade Plan Runtime Extensions

Introduction

The proposal for Chained and efficient upgrades introduced support for upgrading by more than one minor when working with Clusters using managed topologies.

According to the proposal, there are two ways to provide Cluster API the information required to compute the upgrade plan:

• By setting the list of versions in the spec.kubernetesVersions field in the ClusterClass object.
• By calling the runtime hook defined in the spec.upgrade field in the ClusterClass object.

This document defines the hook for the second option and provides recommendations on how to implement it.

• Implementing Upgrade Plan Runtime Extensions
  • Introduction
  • Guidelines
  • Definitions
    • GenerateUpgradePlan

Guidelines

All guidelines defined in Implementing Runtime Extensions apply to the implementation of Runtime Extensions for upgrade plan hooks as well.

In summary, Runtime Extensions are components that should be designed, written and deployed with great caution given that they can affect the proper functioning of the Cluster API runtime. A poorly implemented Runtime Extension could potentially block upgrades.

Following recommendations are especially relevant:

• Idempotence
• Deterministic result
• Error messages
• Error management
• Avoid dependencies

Definitions

For additional details about the OpenAPI spec of the upgrade plan hooks, please download the runtime-sdk-openapi.yaml file and then open it from the Swagger UI.

GenerateUpgradePlan

The GenerateUpgradePlan hook is called every time Cluster API is required to compute the upgrade plan.

Notably, during an upgrade, the upgrade plan is recomputed several times, ideally once each time the upgrade plan completes a step, but the number of calls might be higher depending on e.g. the duration of the upgrade.

Example Request:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: GenerateUpgradePlanRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
    name: test-cluster
    namespace: test-ns
  spec:
    ...
  status:
    ...
fromControlPlaneKubernetesVersion: "v1.29.0"
fromWorkersKubernetesVersion: "v1.29.0"
toKubernetesVersion: "v1.33.0"

Example Response:

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: GenerateUpgradePlanResponse
status: Success # or Failure
message: "error message if status == Failure"
controlPlaneUpgrades:
- version: v1.30.0
- version: v1.31.0
- version: v1.32.3
- version: v1.33.0

Note: in this case the system will infer the list of intermediate version for workers from the list of control plane versions, taking care of performing the minimum number of workers upgrade by taking into account the Kubernetes version skew policy.

Implementers of this runtime extension can also address more sophisticated use cases by computing the response in different ways, e.g.

• Go through more patch release for a minor if necessary, e.g., v1.30.0 -> v1.30.1 -> etc.

  ...
  controlPlaneUpgrades:
  - version: v1.30.0
  - version: v1.30.1
  - ...
  

Note: in this case the system will infer the list of intermediate version for workers from the list of control plane versions, taking care of performing the minimum number of workers upgrade by taking into account the Kubernetes version skew policy.

• Force workers to upgrade to specific versions, e.g., force workers upgrade to v1.30.0 when doing v1.29.0 -> v1.32.3 (in this example, worker upgrade to 1.30.0 is not required by the Kubernetes version skew policy, so it would be skipped under normal circumstances).

  ...
  controlPlaneUpgrades:
  - version: v1.30.0
  - version: v1.31.0
  - version: v1.32.3
  workersUpgrades:
  - version: v1.30.0
  - version: v1.32.3
  

Note: in this case the system will take into consideration the provided workersUpgrades, and validated it is consistent with controlPlaneUpgrades and also compliant with the Kubernetes version skew policy.

• Force workers to upgrade to all the intermediate steps (opt out from efficient upgrades).

  ...
  controlPlaneUpgrades:
  - version: v1.30.0
  - version: v1.31.0
  - version: v1.32.3
  workersUpgrades:
  - version: v1.30.0
  - version: v1.31.0
  - version: v1.32.3
  

Note: in this case the system will take into consideration the provided workersUpgrades, and validate it is consistent with controlPlaneUpgrades and also compliant with the Kubernetes version skew policy.

In all the cases above, the GenerateUpgradePlanResponse content must comply the following validation rules:

• controlPlaneUpgrades is the list of version upgrade steps for the control plane; it must be always specified unless the control plane is already at the target version.

  • there should be at least one version for every minor between fromControlPlaneKubernetesVersion (excluded) and toKubernetesVersion (included).
  • each version must be:
    • greater than fromControlPlaneKubernetesVersion (or with a different build number)
    • greater than the previous version in the list (or with a different build number)
    • less or equal to toKubernetesVersion (or with a different build number)
    • the last version in the plan must be equal to toKubernetesVersion

• workersUpgrades is the list of version upgrade steps for the workers.

  • In case the upgrade plan for workers will be left to empty, the system will automatically determine the minimal number of workers upgrade steps, thus minimizing impact on workloads and reducing the overall upgrade time.
  • If instead for any reason a custom upgrade plan for workers is required, workersUpgrades should be set and the following rules apply to each version in the list. More specifically, each version must be:
    • equal to fromControlPlaneKubernetesVersion or to one of the versions in the control plane upgrade plan.
    • greater than fromWorkersKubernetesVersion (or with a different build number)
    • greater than the previous version in the list (or with a different build number)
    • less or equal to the toKubernetesVersion (or with a different build number)
    • in case of versions with the same major/minor/patch version but different build number, also the order of those versions must be the same for control plane and worker upgrade plan.
    • the last version in the plan must be equal to toKubernetesVersion
    • the upgrade plan must have all the intermediate version which workers must go through to avoid breaking rules defining the max version skew between control plane and workers.

Deploy Runtime Extensions

Cluster API requires that each Runtime Extension must be deployed using an endpoint accessible from the Cluster API controllers. The recommended deployment model is to deploy a Runtime Extension in the management cluster by:

• Packing the Runtime Extension in a container image.
• Using a Kubernetes Deployment to run the above container inside the Management Cluster.
• Using a Cluster IP Service to make the Runtime Extension instances accessible via a stable DNS name.
• Using a cert-manager generated Certificate to protect the endpoint.
• Register the Runtime Extension using ExtensionConfig.

For an example, please see our test extension which follows, as closely as possible, the kubebuilder setup used for controllers in Cluster API.

There are a set of important guidelines that must be considered while choosing the deployment method:

Availability

It is recommended that Runtime Extensions should leverage some form of load-balancing, to provide high availability and performance benefits. You can run multiple Runtime Extension servers behind a Kubernetes Service to leverage the load-balancing that services support.

Identity and access management

The security model for each Runtime Extension should be carefully defined, similar to any other application deployed in the Cluster. If the Runtime Extension requires access to the apiserver the deployment must use a dedicated service account with limited RBAC permission. Otherwise no service account should be used.

On top of that, the container image for the Runtime Extension should be carefully designed in order to avoid privilege escalation (e.g using distroless base images). The Pod spec in the Deployment manifest should enforce security best practices (e.g. do not use privileged pods).

Alternative deployments methods

Alternative deployment methods can be used as long as the HTTPs endpoint is accessible, like e.g.:

• deploying the HTTPS Server as a part of another component, e.g. a controller.
• deploying the HTTPS Server outside the Management Cluster.

In those cases recommendations about availability and identity and access management still apply.

Experimental Feature: Ignition Bootstrap Config (alpha)

The default configuration engine for bootstrapping workload cluster machines is cloud-init. Ignition is an alternative engine used by Linux distributions such as Flatcar Container Linux and Fedora CoreOS and therefore should be used when choosing an Ignition-based distribution as the underlying OS for workload clusters.

This guide explains how to deploy an AWS workload cluster using Ignition.

Prerequisites

• kubectl installed locally
• clusterawsadm installed locally - download from the releases page of the AWS provider
• kind and Docker installed locally (when using kind to create a management cluster)

Configure a management cluster

Follow this section of the quick start guide to deploy a Kubernetes cluster or connect to an existing one.

Follow this section of the quick start guide to install clusterctl.

Initialize the management cluster

Before workload clusters can be deployed, Cluster API components must be deployed to the management cluster.

Initialize the management cluster:

export AWS_REGION=us-east-1
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# Workload clusters need to call the AWS API as part of their normal operation.
# The following command creates a CloudFormation stack which provisions the
# necessary IAM resources to be used by workload clusters.
clusterawsadm bootstrap iam create-cloudformation-stack

# The management cluster needs to call the AWS API in order to manage cloud
# resources for workload clusters. The following command tells clusterctl to
# store the AWS credentials provided before in a Kubernetes secret where they
# can be retrieved by the AWS provider running on the management cluster.
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)

# Enable the feature gates controlling Ignition bootstrap.
export EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true # Used by the kubeadm bootstrap provider
export EXP_BOOTSTRAP_FORMAT_IGNITION=true # Used by the AWS provider

# Initialize the management cluster.
clusterctl init --infrastructure aws

Generate a workload cluster configuration

# Deploy the workload cluster in the following AWS region.
export AWS_REGION=us-east-1

# Authorize the following SSH public key on cluster nodes.
export AWS_SSH_KEY_NAME=my-key

# Ignition bootstrap data needs to be stored in an S3 bucket so that nodes can
# read them at boot time. Store Ignition bootstrap data in the following bucket.
export AWS_S3_BUCKET_NAME=my-bucket

# Set the EC2 machine size for controllers and workers.
export AWS_CONTROL_PLANE_MACHINE_TYPE=t3a.small
export AWS_NODE_MACHINE_TYPE=t3a.small

clusterctl generate cluster ignition-cluster \
    --from https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/main/templates/cluster-template-flatcar.yaml \
    --kubernetes-version v1.28.0 \
    --worker-machine-count 2 \
    > ignition-cluster.yaml

NOTE: Only certain Kubernetes versions have pre-built Kubernetes AMIs. See list of published pre-built Kubernetes AMIs.

Apply the workload cluster

kubectl apply -f ignition-cluster.yaml

Wait for the control plane of the workload cluster to become initialized:

kubectl get kubeadmcontrolplane ignition-cluster-control-plane

This could take a while. When the control plane is initialized, the INITIALIZED field should be true:

NAME                             CLUSTER            INITIALIZED   API SERVER AVAILABLE   REPLICAS   READY   UPDATED   UNAVAILABLE   AGE    VERSION
ignition-cluster-control-plane   ignition-cluster   true                                 1                  1         1             7m7s   v1.22.2

Connect to the workload cluster

Generate a kubeconfig for the workload cluster:

clusterctl get kubeconfig ignition-cluster > ./kubeconfig

Set kubectl to use the generated kubeconfig:

export KUBECONFIG=$(pwd)/kubeconfig

Verify connectivity with the workload cluster’s API server:

kubectl cluster-info

Sample output:

Kubernetes control plane is running at https://ignition-cluster-apiserver-284992524.us-east-1.elb.amazonaws.com:6443
CoreDNS is running at https://ignition-cluster-apiserver-284992524.us-east-1.elb.amazonaws.com:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Deploy a CNI plugin

A CNI plugin must be deployed to the workload cluster for the cluster to become ready. We use Calico here, however other CNI plugins could be used, too.

kubectl apply -f https://docs.projectcalico.org/v3.20/manifests/calico.yaml

Ensure all cluster nodes become ready:

kubectl get nodes

Sample output:

NAME                                            STATUS   ROLES                  AGE   VERSION
ip-10-0-122-154.us-east-1.compute.internal   Ready    control-plane,master   14m   v1.22.2
ip-10-0-127-59.us-east-1.compute.internal    Ready    <none>                 13m   v1.22.2
ip-10-0-89-169.us-east-1.compute.internal    Ready    <none>                 13m   v1.22.2

Clean up

Delete the workload cluster (from a shell connected to the management cluster):

kubectl delete cluster ignition-cluster

Caveats

Supported infrastructure providers

Cluster API has multiple infrastructure providers which can be used to deploy workload clusters.

The following infrastructure providers already have Ignition support:

• AWS

Ignition support will be added to more providers in the future.

Running multiple providers

Cluster API supports running multiple infrastructure/bootstrap/control plane providers on the same management cluster. It’s highly recommended to rely on clusterctl init command in this case. clusterctl will help ensure that all providers support the same API Version of Cluster API (contract).

Verification of CAPI artifacts

Requirements

You will need to have the following tools installed:

• cosign (install guide)
• jq (download jq)

CAPI Images

Each release of the Cluster API project includes the following container images:

• cluster-api-controller
• kubeadm-bootstrap-controller
• kubeadm-control-plane-controller
• clusterctl

Verifying Image Signatures

All of the four images are hosted by registry.k8s.io. In order to verify the authenticity of the images, you can use cosign verify command with the appropriate image name and version:

$ cosign verify registry.k8s.io/cluster-api/cluster-api-controller:v1.5.0 --certificate-identity krel-trust@k8s-releng-prod.iam.gserviceaccount.com --certificate-oidc-issuer https://accounts.google.com | jq .

Verification for registry.k8s.io/cluster-api/cluster-api-controller:v1.5.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
[
  {
    "critical": {
      "identity": {
        "docker-reference": "registry.k8s.io/cluster-api/cluster-api-controller"
      },
      "image": {
        "docker-manifest-digest": "sha256:f34016d3a494f9544a16137c9bba49d8756c574a0a1baf96257903409ef82f77"
      },
      "type": "cosign container image signature"
    },
    "optional": {
      "1.3.6.1.4.1.57264.1.1": "https://accounts.google.com",
      "Bundle": {
        "SignedEntryTimestamp": "MEYCIQDtxr/v3uRl2QByVfYo1oopruADSaH3E4wThpmkibJs8gIhAIe0odbk99na5GBdYGjJ6IwpFzhlTlicgWOrsgxZH8LC",
        "Payload": {
          "body": "eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiIzMDMzNzY0MTQwZmI2OTE5ZjRmNDg2MDgwMDZjYzY1ODU2M2RkNjE0NWExMzVhMzE5MmQyYTAzNjE1OTRjMTRlIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FUUNJQ3RtcGdHN3RDcXNDYlk0VlpXNyt6Rm5tYWYzdjV4OTEwcWxlWGppdTFvbkFpQS9JUUVSSDErdit1a0hrTURSVnZnN1hPdXdqTTN4REFOdEZyS3NUMHFzaUE9PSIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCRFJWSlVTVVpKUTBGVVJTMHRMUzB0Q2sxSlNVTTJha05EUVc1SFowRjNTVUpCWjBsVldqYzNUbGRSV1VacmQwNTVRMk13Y25GWWJIcHlXa3RyYURjMGQwTm5XVWxMYjFwSmVtb3dSVUYzVFhjS1RucEZWazFDVFVkQk1WVkZRMmhOVFdNeWJHNWpNMUoyWTIxVmRWcEhWakpOVWpSM1NFRlpSRlpSVVVSRmVGWjZZVmRrZW1SSE9YbGFVekZ3WW01U2JBcGpiVEZzV2tkc2FHUkhWWGRJYUdOT1RXcE5kMDU2U1RGTlZHTjNUa1JOTlZkb1kwNU5hazEzVG5wSk1VMVVZM2hPUkUwMVYycEJRVTFHYTNkRmQxbElDa3R2V2tsNmFqQkRRVkZaU1V0dldrbDZhakJFUVZGalJGRm5RVVZ4VEdveFJsSmhLM2RZTUVNd0sxYzFTVlZWUW14UmRsWkNWM2xLWTFRcmFWaERjV01LWTA4d1prVmpNV2s0TVUxSFQwRk1lVXB2UXpGNk5TdHVaRGxFUnpaSGNFSmpOV0ZJYXpoU1QxaDBOV2h6U21wa1VVdFBRMEZhUVhkblowZE5UVUUwUndwQk1WVmtSSGRGUWk5M1VVVkJkMGxJWjBSQlZFSm5UbFpJVTFWRlJFUkJTMEpuWjNKQ1owVkdRbEZqUkVGNlFXUkNaMDVXU0ZFMFJVWm5VVlYxTVRoMENqWjVWMWxNVlU5RVR5dEVjek52VVU1RFNsYzNZMUJWZDBoM1dVUldVakJxUWtKbmQwWnZRVlV6T1ZCd2VqRlphMFZhWWpWeFRtcHdTMFpYYVhocE5Ga0tXa1E0ZDFGQldVUldVakJTUVZGSUwwSkVXWGRPU1VWNVlUTktiR0pETVRCamJsWjZaRVZDY2s5SVRYUmpiVlp6V2xjMWJreFlRbmxpTWxGMVlWZEdkQXBNYldSNldsaEtNbUZYVG14WlYwNXFZak5XZFdSRE5XcGlNakIzUzFGWlMwdDNXVUpDUVVkRWRucEJRa0ZSVVdKaFNGSXdZMGhOTmt4NU9XaFpNazUyQ21SWE5UQmplVFZ1WWpJNWJtSkhWWFZaTWpsMFRVTnpSME5wYzBkQlVWRkNaemM0ZDBGUlowVklVWGRpWVVoU01HTklUVFpNZVRsb1dUSk9kbVJYTlRBS1kzazFibUl5T1c1aVIxVjFXVEk1ZEUxSlIwdENaMjl5UW1kRlJVRmtXalZCWjFGRFFraDNSV1ZuUWpSQlNGbEJNMVF3ZDJGellraEZWRXBxUjFJMFl3cHRWMk16UVhGS1MxaHlhbVZRU3pNdmFEUndlV2RET0hBM2J6UkJRVUZIU21wblMxQmlkMEZCUWtGTlFWSjZRa1pCYVVKSmJXeGxTWEFyTm05WlpVWm9DbWRFTTI1Uk5sazBSV2g2U25SVmMxRTRSSEJrWTFGeU5FSk1XRE41ZDBsb1FVdFhkV05tYmxCUk9GaExPWGRZYkVwcVNWQTBZMFpFT0c1blpIazRkV29LYldreGN6RkRTamczTW1zclRVRnZSME5EY1VkVFRUUTVRa0ZOUkVFeVkwRk5SMUZEVFVoaU9YRjBSbGQxT1VGUU1FSXpaR3RKVkVZNGVrazRZVEkxVUFwb2IwbFBVVlJLVWxKeGFsVmlUMkUyVnpOMlRVZEJOWFpKTlZkVVJqQkZjREZwTWtGT2QwbDNSVko0TW5ocWVtWjNjbmRPYmxoUVpEQjRjbmd3WWxoRENtUmpOV0Z4WWxsWlVsRXdMMWhSVVdONFRFVnRkVGwzUnpGRlYydFNNWE01VEdaUGVHZDNVMjRLTFMwdExTMUZUa1FnUTBWU1ZFbEdTVU5CVkVVdExTMHRMUW89In19fX0=",
          "integratedTime": 1690304684,
          "logIndex": 28719030,
          "logID": "c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"
        }
      },
      "Issuer": "https://accounts.google.com",
      "Subject": "krel-trust@k8s-releng-prod.iam.gserviceaccount.com",
      "org.kubernetes.kpromo.version": "kpromo-v4.0.3-5-ge99897c"
    }
  }
]

Diagnostics

Introduction

With CAPI v1.6 we introduced new flags to allow serving metrics, the pprof endpoint and an endpoint to dynamically change log levels securely in production.

This feature is enabled by default via:

          args:
            - "--diagnostics-address=${CAPI_DIAGNOSTICS_ADDRESS:=:8443}"

As soon as the feature is enabled the metrics endpoint is served via https and protected via authentication and authorization. This works the same way as metrics in core Kubernetes components: Metrics in Kubernetes.

To continue serving metrics via http the following configuration can be used:

          args:
            - "--diagnostics-address=localhost:8080"
            - "--insecure-diagnostics"

The same can be achieved via clusterctl:

export CAPI_DIAGNOSTICS_ADDRESS="localhost:8080"
export CAPI_INSECURE_DIAGNOSTICS="true"
clusterctl init ...

Note: If insecure serving is configured the pprof and log level endpoints are disabled for security reasons.

Scraping metrics

A ServiceAccount token is now required to scrape metrics. The corresponding ServiceAccount needs permissions on the /metrics path. This can be achieved e.g. by following the Kubernetes documentation.

via Prometheus

With the Prometheus Helm chart it is as easy as using the following config for the Prometheus job scraping the Cluster API controllers:

    scheme: https
    authorization:
      type: Bearer
      credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      # The diagnostics endpoint is using a self-signed certificate, so we don't verify it.
      insecure_skip_verify: true

For more details please see our Prometheus development setup: Prometheus

Note: The Prometheus Helm chart deploys the required ClusterRole out-of-the-box.

via kubectl

First deploy the following RBAC configuration:

cat << EOT | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: default-metrics
rules:
- nonResourceURLs:
  - "/metrics"
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: default-metrics
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
EOT

Then let’s open a port-forward, create a ServiceAccount token and scrape the metrics:

# Terminal 1
kubectl -n capi-system port-forward deployments/capi-controller-manager 8443

# Terminal 2
TOKEN=$(kubectl create token default)
curl https://localhost:8443/metrics --header "Authorization: Bearer $TOKEN" -k

Collecting profiles

via Parca

Parca can be used to continuously scrape profiles from CAPI providers. For more details please see our Parca development setup: parca

via kubectl

First deploy the following RBAC configuration:

cat << EOT | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: default-pprof
rules:
- nonResourceURLs:
  - "/debug/pprof/*"
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-pprof
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: default-pprof
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
EOT

Then let’s open a port-forward, create a ServiceAccount token and scrape the profile:

# Terminal 1
kubectl -n capi-system port-forward deployments/capi-controller-manager 8443

# Terminal 2
TOKEN=$(kubectl create token default)

# Get a goroutine dump
curl "https://localhost:8443/debug/pprof/goroutine?debug=2" --header "Authorization: Bearer $TOKEN" -k > ./goroutine.txt

# Get a profile
curl "https://localhost:8443/debug/pprof/profile?seconds=10" --header "Authorization: Bearer $TOKEN" -k > ./profile.out
go tool pprof -http=:8080 ./profile.out

Changing the log level

via kubectl

First deploy the following RBAC configuration:

cat << EOT | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: default-loglevel
rules:
- nonResourceURLs:
  - "/debug/flags/v"
  verbs:
  - put
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-loglevel
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: default-loglevel
subjects:
- kind: ServiceAccount
  name: default
  namespace: default
EOT

Then let’s open a port-forward, create a ServiceAccount token and change the log level to 8:

# Terminal 1
kubectl -n capi-system port-forward deployments/capi-controller-manager 8443

# Terminal 2
TOKEN=$(kubectl create token default)
curl "https://localhost:8443/debug/flags/v" --header "Authorization: Bearer $TOKEN" -X PUT -d '8' -k

ClusterResourceSet (GA)

The ClusterResourceSet feature is introduced to provide a way to automatically apply a set of resources (such as CNI/CSI) defined by users to matching newly-created/existing clusters. ClusterResourceSet provides a basic solution for installing & managing resources, while for advanced use cases an addon provider must be used.

ClusterResourceSet is namespace-scoped, all resources and clusters referenced in the ClusterResourceSet spec need to be in the same namespace as the ClusterResourceSet.

More details on ClusterResourceSet can be found at: ClusterResourceSet CAEP

Example

Suppose you want to automatically install the relevant external cloud provider on all workload clusters. This can be accomplished by labeling the clusters with the specific cloud (e.g. AWS, GCP or OpenStack) and then creating a ClusterResourceSet for each. For example, you could have the following for OpenStack:

apiVersion: addons.cluster.x-k8s.io/v1beta2
kind: ClusterResourceSet
metadata:
  name: cloud-provider-openstack
  namespace: default
spec:
  strategy: Reconcile
  clusterSelector:
    matchLabels:
      cloud: openstack
  resources:
    - name: cloud-provider-openstack
      kind: ConfigMap
    - name: cloud-config
      kind: Secret

This ClusterResourceSet would apply the content of the Secret cloud-config and of the ConfigMap cloud-provider-openstack in all workload clusters with the label cloud=openstack. Suppose you have the file cloud.conf that should be included in the Secret and cloud-provider-openstack.yaml that should be in the ConfigMap. The Secret and ConfigMap can then be created in the following way:

kubectl create secret generic cloud-config --from-file=cloud.conf --type=addons.cluster.x-k8s.io/resource-set
kubectl create configmap cloud-provider-openstack --from-file=cloud-provider-openstack.yaml

Note that it is required that the Secret has the type addons.cluster.x-k8s.io/resource-set for it to be picked up.

Update from ApplyOnce to Reconcile

The strategy field is immutable so existing CRS can’t be updated directly. However, CAPI won’t delete the managed resources in the target cluster when the CRS is deleted. So if you want to start using the Reconcile strategy, delete your existing CRS and create it again with the updated strategy.

Security Guidelines

This section provides security guidelines useful to provision clusters which are secure by default to follow the secure defaults guidelines for cloud native apps.

Pod Security Standards

Pod Security Admission allows applying Pod Security Standards during creation of pods at the cluster level.

The flavor development-topology for the Docker provider used in Quick Start already includes a basic Pod Security Standard configuration. It is using ClusterClass variables and patches to inject the configuration.

Adding a basic Pod Security Standards configuration to a ClusterClass

By adding the following variables and patches Pod Security Standards can be added to every ClusterClass which references a Kubeadm based control plane.

Adding the variables to a ClusterClass

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
spec:
  variables:
  - name: podSecurityStandard
    required: false
    schema:
      openAPIV3Schema:
        type: object
        properties: 
          enabled: 
            type: boolean
            default: true
            description: "enabled enables the patches to enable Pod Security Standard via AdmissionConfiguration."
          enforce:
            type: string
            default: "baseline"
            description: "enforce sets the level for the enforce PodSecurityConfiguration mode. One of privileged, baseline, restricted."
            pattern: "privileged|baseline|restricted"
          audit:
            type: string
            default: "restricted"
            description: "audit sets the level for the audit PodSecurityConfiguration mode. One of privileged, baseline, restricted."
            pattern: "privileged|baseline|restricted"
          warn:
            type: string
            default: "restricted"
            description: "warn sets the level for the warn PodSecurityConfiguration mode. One of privileged, baseline, restricted."
            pattern: "privileged|baseline|restricted"
  ...

• The version field in Pod Security Admission Config defaults to latest.
• The kube-system namespace is exempt from Pod Security Standards enforcement, because it runs control-plane pods that need higher privileges.

Adding the patches to a ClusterClass

The following snippet contains the patch to be added to the ClusterClass.

Due to limitations of ClusterClass with patches there are two versions for this patch.

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
spec:
  ...
  patches:
  - name: podSecurityStandard
    description: "Adds an admission configuration for PodSecurity to the kube-apiserver."
    definitions:
    - selector:
        apiVersion: controlplane.cluster.x-k8s.io/v1beta1
        kind: KubeadmControlPlaneTemplate
        matchResources:
          controlPlane: true
      jsonPatches:
      - op: add
        path: "/spec/template/spec/kubeadmConfigSpec/clusterConfiguration/apiServer/extraArgs"
        value:
          admission-control-config-file: "/etc/kubernetes/kube-apiserver-admission-pss.yaml"
      - op: add
        path: "/spec/template/spec/kubeadmConfigSpec/clusterConfiguration/apiServer/extraVolumes/-"
        value:
          name: admission-pss
          hostPath: /etc/kubernetes/kube-apiserver-admission-pss.yaml
          mountPath: /etc/kubernetes/kube-apiserver-admission-pss.yaml
          readOnly: true
          pathType: "File"
      - op: add
        path: "/spec/template/spec/kubeadmConfigSpec/files/-"
        valueFrom:
          template: |
            content: |
              apiVersion: apiserver.config.k8s.io/v1
              kind: AdmissionConfiguration
              plugins:
              - name: PodSecurity
                configuration:
                  apiVersion: pod-security.admission.config.k8s.io/v1{{ if semverCompare "< v1.25" .builtin.controlPlane.version }}beta1{{ end }}
                  kind: PodSecurityConfiguration
                  defaults:
                    enforce: "{{ .podSecurityStandard.enforce }}"
                    enforce-version: "latest"
                    audit: "{{ .podSecurityStandard.audit }}"
                    audit-version: "latest"
                    warn: "{{ .podSecurityStandard.warn }}"
                    warn-version: "latest"
                  exemptions:
                    usernames: []
                    runtimeClasses: []
                    namespaces: [kube-system]
            path: /etc/kubernetes/kube-apiserver-admission-pss.yaml
    enabledIf: "{{ .podSecurityStandard.enabled }}"
...

apiVersion: cluster.x-k8s.io/v1beta2
kind: ClusterClass
spec:
  ...
  patches:
  - name: podSecurityStandard
    description: "Adds an admission configuration for PodSecurity to the kube-apiserver."
    definitions:
    - selector:
        apiVersion: controlplane.cluster.x-k8s.io/v1beta1
        kind: KubeadmControlPlaneTemplate
        matchResources:
          controlPlane: true
      jsonPatches:
      - op: add
        path: "/spec/template/spec/kubeadmConfigSpec/clusterConfiguration/apiServer/extraArgs"
        value:
          admission-control-config-file: "/etc/kubernetes/kube-apiserver-admission-pss.yaml"
      - op: add
        path: "/spec/template/spec/kubeadmConfigSpec/clusterConfiguration/apiServer/extraVolumes"
        value:
        - name: admission-pss
          hostPath: /etc/kubernetes/kube-apiserver-admission-pss.yaml
          mountPath: /etc/kubernetes/kube-apiserver-admission-pss.yaml
          readOnly: true
          pathType: "File"
      - op: add
        path: "/spec/template/spec/kubeadmConfigSpec/files"
        valueFrom:
          template: |
            - content: |
                apiVersion: apiserver.config.k8s.io/v1
                kind: AdmissionConfiguration
                plugins:
                - name: PodSecurity
                  configuration:
                    apiVersion: pod-security.admission.config.k8s.io/v1{{ if semverCompare "< v1.25" .builtin.controlPlane.version }}beta1{{ end }}
                    kind: PodSecurityConfiguration
                    defaults:
                      enforce: "{{ .podSecurityStandard.enforce }}"
                      enforce-version: "latest"
                      audit: "{{ .podSecurityStandard.audit }}"
                      audit-version: "latest"
                      warn: "{{ .podSecurityStandard.warn }}"
                      warn-version: "latest"
                    exemptions:
                      usernames: []
                      runtimeClasses: []
                      namespaces: [kube-system]
              path: /etc/kubernetes/kube-apiserver-admission-pss.yaml
    enabledIf: "{{ .podSecurityStandard.enabled }}"
...

Create a secure Cluster using the ClusterClass

After adding the variables and patches the Pod Security Standards would be applied by default. It is also possible to disable this patch or configure different levels for the configuration using variables.

apiVersion: cluster.x-k8s.io/v1beta2
kind: Cluster
metadata:
  name: "my-cluster"
spec:
  ...
  topology:
    ...
    classRef:
      name: my-secure-cluster-class
    variables:
    - name: podSecurityStandard
      value: 
        enabled: true
        enforce: "restricted"

Security Guidelines for Cluster API Users

This document compiles security best practices for using Cluster API. These guidelines are based on the Cluster API Security Self-Assessment conducted by the Kubernetes SIG Security. We recommend that organizations adapt these guidelines to their specific infrastructure and security requirements to ensure safe operations.

Comprehensive auditing

To ensure comprehensive auditing, the following components require audit configuration:

• Cluster-level Auditing

  • Auditing on the management cluster
  • API server auditing for all workload clusters

• Node/VM-level Auditing

  • Audit KubeConfig files access that are located on the node
  • Audit access or edits to CA private keys and cert files located on the node

• Cloud Provider Auditing

  • Cloud API auditing to log all actions performed using cloud credentials

After configuring these audit sources, centralize the logs using aggregation tools and implement real-time monitoring and alerting to detect suspicious activities and security incidents.

Use least privileges

To minimize security risks related to cloud provider access, create dedicated cloud credentials that have only the necessary permissions to manage the lifecycle of a cluster. Avoid using administrative or root accounts for Cluster API operations, and use separate credentials for different purposes such as management cluster versus workload clusters.

Limit access

Implement access restrictions to protect cluster infrastructure.

Control Plane Protection

Limit who can create pods on control plane nodes through multiple methods:

• Taints and Tolerations: Apply NoSchedule taints to control plane nodes to prevent general workload scheduling. See Kubernetes Taints and Tolerations documentation
• RBAC Policies: Restrict pod creation permissions using Role-Based Access Control. See Kubernetes RBAC documentation
• Admission Controllers: Implement admission webhooks to enforce pod placement policies. See Dynamic Admission Control

SSH Access

Disable or restrict SSH access to nodes in a cluster to prevent unauthorized modifications and access to sensitive files.

Second pair of eyes

Implement a review process where at least two people must approve privileged actions such as creating, deleting, or updating clusters. GitOps provides an effective way to enforce this requirement through pull request workflows, where changes to cluster configurations must be reviewed and approved by another team member before being merged and applied to the infrastructure.

Implement comprehensive alerting

Configure alerts in the centralized audit log system to detect security incidents and resource anomalies.

Security Event Monitoring

• Alert when cluster API components are modified, restarted, or experience unexpected state changes
• Monitor and alert on unauthorized changes to sensitive files on machine images
• Alert on unexpected machine restarts or shutdowns
• Monitor deletion or modification of Elastic Load Balancers (ELB) for API servers

Resource Activity Monitoring

• Alert on all cloud resource creation, update, and deletion activities
• Identify anomalous patterns such as mass resource creation or deletion
• Monitor for resources created outside expected boundaries

Resource Limit Monitoring

• Alert when the number of clusters approaches or exceeds defined soft limits
• Monitor node creation rates and alert when approaching capacity limits
• Track usage against cloud provider quotas and organizational limits
• Alert on excessive API calls or resource creation requests

Cluster isolation and segregation

Implement multiple layers of isolation to prevent privilege escalation from workload clusters to management cluster.

Account/Subscription Separation

Separate workload clusters into different AWS accounts or Azure subscriptions, and use dedicated accounts for management cluster and production workloads. This approach provides a strong security boundary at the cloud provider level.

Network Boundaries

Separate workload and management clusters at the network level through VPC boundaries. Use dedicated VPC/VNet for each cluster type to prevent lateral movement between clusters.

Certificate Authority Isolation

Do not build a chain of trust for cluster CAs. Each cluster must have its own independent CA to ensure that workload cluster CA compromise does not provide access to the management cluster. See Kubernetes PKI certificates and requirements for best practices.

Prevent runtime updates

Implement controls to prevent tampering of machine images at runtime. Disable or restrict updates to machine images at runtime and prevent unauthorized modifications through SSH access restrictions. Following immutable infrastructure practices ensures that any changes require deploying new images rather than modifying running systems.

Overview of clusterctl

The clusterctl CLI tool handles the lifecycle of a Cluster API management cluster.

The clusterctl command line interface is specifically designed for providing a simple “day 1 experience” and a quick start with Cluster API. It automates fetching the YAML files defining provider components and installing them.

Additionally it encodes a set of best practices in managing providers, that helps the user in avoiding mis-configurations or in managing day 2 operations such as upgrades.

Below you can find a list of main clusterctl commands:

• clusterctl init Initialize a management cluster.
• clusterctl upgrade plan Provide a list of recommended target versions for upgrading Cluster API providers in a management cluster.
• clusterctl upgrade apply Apply new versions of Cluster API core and providers in a management cluster.
• clusterctl delete Delete one or more providers from the management cluster.
• clusterctl generate cluster Generate templates for creating workload clusters.
• clusterctl generate yaml Process yaml using clusterctl’s yaml processor.
• clusterctl get kubeconfig Gets the kubeconfig file for accessing a workload cluster.
• clusterctl move Move Cluster API objects and all their dependencies between management clusters.
• clusterctl alpha rollout Manages the rollout of Cluster API resources. For example: MachineDeployments.

For the full list of clusterctl commands please refer to commands.

Avoiding GitHub rate limiting

While using providers hosted on GitHub, clusterctl is calling GitHub API which are rate limited; for normal usage free tier is enough but when using clusterctl extensively users might hit the rate limit.

To avoid rate limiting for the public repos set the GITHUB_TOKEN environment variable. To generate a token follow this documentation. The token only needs repo scope for clusterctl.

Per default clusterctl will use a go proxy to detect the available versions to prevent additional API calls to the GitHub API. It is possible to configure the go proxy url using the GOPROXY variable as for go itself (defaults to https://proxy.golang.org). To immediately fallback to the GitHub client and not use a go proxy, the environment variable could get set to GOPROXY=off or GOPROXY=direct. If a provider does not follow Go’s semantic versioning, clusterctl may fail when detecting the correct version. In such cases, disabling the go proxy functionality via GOPROXY=off should be considered.

Installing clusterctl

Instructions are available in the Quick Start.

clusterctl commands

clusterctl init

The clusterctl init command installs the Cluster API components and transforms the Kubernetes cluster into a management cluster.

This document provides more detail on how clusterctl init works and on the supported options for customizing your management cluster.

Defining the management cluster

The clusterctl init command accepts in input a list of providers to install.

Automatically installed providers

The clusterctl init command automatically adds the cluster-api core provider, the kubeadm bootstrap provider, and the kubeadm control-plane provider to the list of providers to install. This allows users to use a concise command syntax for initializing a management cluster. For example, to get a fully operational management cluster with the aws infrastructure provider, the cluster-api core provider, the kubeadm bootstrap, and the kubeadm control-plane provider, use the command:

clusterctl init --infrastructure aws

Provider version

The clusterctl init command by default installs the latest version available for each selected provider.

Target namespace

The clusterctl init command by default installs each provider in the default target namespace defined by each provider, e.g. capi-system for the Cluster API core provider.

See the provider documentation for more details.

Provider repositories

To access provider specific information, such as the components YAML to be used for installing a provider, clusterctl init accesses the provider repositories, that are well-known places where the release assets for a provider are published.

Per default clusterctl will use a go proxy to detect the available versions to prevent additional API calls to the GitHub API. It is possible to configure the go proxy url using the GOPROXY variable as for go itself (defaults to https://proxy.golang.org). To immediately fallback to the GitHub client and not use a go proxy, the environment variable could get set to GOPROXY=off or GOPROXY=direct. If a provider does not follow Go’s semantic versioning, clusterctl may fail when detecting the correct version. In such cases, disabling the go proxy functionality via GOPROXY=off should be considered.

See clusterctl configuration for more info about provider repository configurations.

Variable substitution

Providers can use variables in the components YAML published in the provider’s repository.

During clusterctl init, those variables are replaced with environment variables or with variables read from the clusterctl configuration.

Additional information

When installing a provider, the clusterctl init command executes a set of steps to simplify the lifecycle management of the provider’s components.

• All the provider’s components are labeled, so they can be easily identified in subsequent moments of the provider’s lifecycle, e.g. upgrades.

labels:
- clusterctl.cluster.x-k8s.io: ""
- cluster.x-k8s.io/provider: "<provider-name>"

• An additional Provider object is created in the target namespace where the provider is installed. This object keeps track of the provider version, and other useful information for the inventory of the providers currently installed in the management cluster.

Cert-manager

Cluster API providers require a cert-manager version supporting the cert-manager.io/v1 API to be installed in the cluster.

While doing init, clusterctl checks if there is a version of cert-manager already installed. If not, clusterctl will install a default version (currently cert-manager v1.21.1). See clusterctl configuration for available options to customize this operation.

Avoiding GitHub rate limiting

Follow this

clusterctl generate cluster

The clusterctl generate cluster command returns a YAML template for creating a workload cluster.

For example

clusterctl generate cluster my-cluster --kubernetes-version v1.28.0 --control-plane-machine-count=3 --worker-machine-count=3 > my-cluster.yaml

Generates a YAML file named my-cluster.yaml with a predefined list of Cluster API objects; Cluster, Machines, Machine Deployments, etc. to be deployed in the current namespace (in case, use the --target-namespace flag to specify a different target namespace).

Then, the file can be modified using your editor of choice; when ready, run the following command to apply the cluster manifest.

kubectl apply -f my-cluster.yaml

Selecting the infrastructure provider to use

The clusterctl generate cluster command uses smart defaults in order to simplify the user experience; in the example above, it detects that there is only an aws infrastructure provider in the current management cluster and so it automatically selects a cluster template from the aws provider’s repository.

In case there is more than one infrastructure provider, the following syntax can be used to select which infrastructure provider to use for the workload cluster:

clusterctl generate cluster my-cluster --kubernetes-version v1.28.0 \
    --infrastructure aws > my-cluster.yaml

or

clusterctl generate cluster my-cluster --kubernetes-version v1.28.0 \
    --infrastructure aws:v0.4.1 > my-cluster.yaml

Flavors

The infrastructure provider authors can provide different types of cluster templates, or flavors; use the --flavor flag to specify which flavor to use; e.g.

clusterctl generate cluster my-cluster --kubernetes-version v1.28.0 \
    --flavor high-availability > my-cluster.yaml

Please refer to the providers documentation for more info about available flavors.

Alternative source for cluster templates

clusterctl uses the provider’s repository as a primary source for cluster templates; the following alternative sources for cluster templates can be used as well:

ConfigMaps

Use the --from-config-map flag to read cluster templates stored in a Kubernetes ConfigMap; e.g.

clusterctl generate cluster my-cluster --kubernetes-version v1.28.0 \
    --from-config-map my-templates > my-cluster.yaml

Also following flags are available --from-config-map-namespace (defaults to current namespace) and --from-config-map-key (defaults to template).

GitHub, raw template URL, local file system folder or standard input

Use the --from flag to read cluster templates stored in a GitHub repository, raw template URL, in a local file system folder, or from the standard input; e.g.

clusterctl generate cluster my-cluster --kubernetes-version v1.28.0 \
   --from https://github.com/my-org/my-repository/blob/main/my-template.yaml > my-cluster.yaml

or

clusterctl generate cluster my-cluster --kubernetes-version v1.28.0 \
   --from https://foo.bar/my-template.yaml > my-cluster.yaml

or

clusterctl generate cluster my-cluster --kubernetes-version v1.28.0 \
   --from ~/my-template.yaml > my-cluster.yaml

or

cat ~/my-template.yaml | clusterctl generate cluster my-cluster --kubernetes-version v1.28.0 \
    --from - > my-cluster.yaml

Variables

If the selected cluster template expects some environment variables, the user should ensure those variables are set in advance.

E.g. if the AWS_CREDENTIALS variable is expected for a cluster template targeting the aws infrastructure, you should ensure the corresponding environment variable to be set before executing clusterctl generate cluster.

Please refer to the providers documentation for more info about the required variables or use the clusterctl generate cluster --list-variables flag to get a list of variables names required by a cluster template.

The clusterctl configuration file can be used as alternative to environment variables.

clusterctl generate provider

Generate templates for provider components.

clusterctl fetches the provider components from the provider repository and performs variable substitution.

Variable values are either sourced from the clusterctl config file or from environment variables

Usage: clusterctl generate provider [flags]

Current usage of the command is as follows:

# Generates a yaml file for creating provider with variable values using
# components defined in the provider repository.
clusterctl generate provider --infrastructure aws

# Generates a yaml file for creating provider for a specific version with variable values using
# components defined in the provider repository.
clusterctl generate provider --infrastructure aws:v0.4.1

# Displays information about a specific infrastructure provider.
# If applicable, prints out the list of required environment variables.
clusterctl generate provider --infrastructure aws --describe

# Displays information about a specific version of the infrastructure provider.
clusterctl generate provider --infrastructure aws:v0.4.1 --describe

# Generates a yaml file for creating provider for a specific version.
# No variables will be processed and substituted using this flag
clusterctl generate provider --infrastructure aws:v0.4.1 --raw

clusterctl generate yaml

The clusterctl generate yaml command processes yaml using clusterctl’s yaml processor.

The intent of this command is to allow users who may have specific templates to leverage clusterctl’s yaml processor for variable substitution. For example, this command can be leveraged in local and CI scripts or for development purposes.

clusterctl ships with a simple yaml processor that performs variable substitution that takes into account default values. Under the hood, clusterctl’s yaml processor uses drone/envsubst to replace variables and uses the defaults if necessary.

Variable values are either sourced from the clusterctl config file or from environment variables.

Current usage of the command is as follows:

# Generates a configuration file with variable values using a template from a
# specific URL as well as a GitHub URL.
clusterctl generate yaml --from https://github.com/foo-org/foo-repository/blob/main/cluster-template.yaml

clusterctl generate yaml --from https://foo.bar/cluster-template.yaml

# Generates a configuration file with variable values using
# a template stored locally.
clusterctl generate yaml  --from ~/workspace/cluster-template.yaml

# Prints list of variables used in the local template
clusterctl generate yaml --from ~/workspace/cluster-template.yaml --list-variables

# Prints list of variables from template passed in via stdin
cat ~/workspace/cluster-template.yaml | clusterctl generate yaml --from - --list-variables

# Default behavior for this sub-command is to read from stdin.
# Generate configuration from stdin
cat ~/workspace/cluster-template.yaml | clusterctl generate yaml

clusterctl get kubeconfig

This command prints the kubeconfig of an existing workload cluster into stdout. This functionality is available in clusterctl v0.3.9 or newer.

Examples

Get the kubeconfig of a workload cluster named foo.

clusterctl get kubeconfig foo

Get the kubeconfig of a workload cluster named foo in the namespace bar

clusterctl get kubeconfig foo --namespace bar

Get the kubeconfig of a workload cluster named foo using a specific context bar

clusterctl get kubeconfig foo --kubeconfig-context bar

clusterctl describe cluster

The clusterctl describe cluster command provides an “at a glance” view of a Cluster API cluster designed to help the user in quickly understanding if there are problems and where.

For example clusterctl describe cluster capi-quickstart will provide an output similar to:

The “at a glance” view is based on the idea that clusterctl should avoid overloading the user with information, but instead surface problems, if any.

In practice, if you look at the ControlPlane node, you might notice that the underlying machines are grouped together, because all of them have the same state (Ready equal to True), so it is not necessary to repeat the same information three times.

If this is not the case, and machines have different states, the visualization is going to use different lines:

You might also notice that the visualization does not represent the infrastructure machine or the bootstrap object linked to a machine, unless their state differs from the machine’s state.

Customizing the visualization

By default, the visualization generated by clusterctl describe cluster hides details for the sake of simplicity and shortness. However, if required, the user can ask for showing all the detail:

By using --grouping=false, the user can force the visualization to show all the machines on separated lines, no matter if they have the same state or not:

By using the --echo flag, the user can force the visualization to show infrastructure machines and bootstrap objects linked to machines, no matter if they have the same state or not:

It is also possible to force the visualization to show all the conditions for an object (instead of showing only the ready condition). e.g. with --show-conditions KubeadmControlPlane you get:

Please note that this option is flexible, and you can pass a comma separated list of kind or kind/name for which the command should show all the object’s conditions (use ‘all’ to show conditions for everything).

clusterctl convert

Warning: This command is EXPERIMENTAL and may be removed in a future release!

The clusterctl convert command converts Cluster API resources between API versions.

Usage

clusterctl convert [SOURCE] [flags]

Examples

# Convert from file to stdout
clusterctl convert cluster.yaml

# Convert from stdin to stdout
cat cluster.yaml | clusterctl convert

# Save output to a file
clusterctl convert cluster.yaml --output converted-cluster.yaml

# Explicitly specify target version
clusterctl convert cluster.yaml --to-version v1beta2 --output converted-cluster.yaml

Flags

• --output, -o: Output file path (default: stdout)
• --to-version: Target API version for conversion (default: “v1beta2”)

Scope and Limitations

• Only cluster.x-k8s.io resources are converted - Core CAPI resources like Cluster, MachineDeployment, Machine, etc.
• Other CAPI API groups are passed through unchanged - Infrastructure, bootstrap, and control plane provider resources are not converted
• ClusterClass patches are not converted - Manual intervention required for ClusterClass patch conversions
• Field order may change - YAML field ordering is not preserved in the output
• Comments are removed - YAML comments are stripped during conversion
• API version references are dropped - Except for ClusterClass and external remediation references

clusterctl move

The clusterctl move command allows to move the Cluster API objects defining workload clusters, like e.g. Cluster, Machines, MachineDeployments, etc. from one management cluster to another management cluster.

You can use:

clusterctl move --to-kubeconfig="path-to-target-kubeconfig.yaml"

To move the Cluster API objects existing in the current namespace of the source management cluster; in case if you want to move the Cluster API objects defined in another namespace, you can use the --namespace flag.

The discovery mechanism for determining the objects to be moved is in the provider contract

Pivot

Pivoting is a process for moving the provider components and declared Cluster API resources from a source management cluster to a target management cluster.

This can now be achieved with the following procedure:

1. Use clusterctl init to install the provider components into the target management cluster
2. Use clusterctl move to move the cluster-api resources from a Source Management cluster to a Target Management cluster

Bootstrap & Pivot

The pivot process can be bounded with the creation of a temporary bootstrap cluster used to provision a target Management cluster.

This can now be achieved with the following procedure:

1. Create a temporary bootstrap cluster, e.g. using kind or minikube
2. Use clusterctl init to install the provider components
3. Use clusterctl generate cluster ... | kubectl apply -f - to provision a target management cluster
4. Wait for the target management cluster to be up and running
5. Get the kubeconfig for the new target management cluster
6. Use clusterctl init with the new cluster’s kubeconfig to install the provider components
7. Use clusterctl move to move the Cluster API resources from the bootstrap cluster to the target management cluster
8. Delete the bootstrap cluster

Note: It’s required to have at least one worker node to schedule Cluster API workloads (i.e. controllers). A cluster with a single control plane node won’t be sufficient due to the NoSchedule taint. If a worker node isn’t available, clusterctl init will timeout.

Dry run

With --dry-run option you can dry-run the move action by only printing logs without taking any actual actions. Use log level verbosity -v to see different levels of information.

clusterctl upgrade

The clusterctl upgrade command can be used to upgrade the version of the Cluster API providers (CRDs, controllers) installed into a management cluster.

upgrade plan

The clusterctl upgrade plan command can be used to identify possible targets for upgrades.

clusterctl upgrade plan

Produces an output similar to this:

Checking cert-manager version...
Cert-Manager will be upgraded from "v1.5.0" to "v1.5.3"

Checking new release availability...

Management group: capi-system/cluster-api, latest release available for the v1beta1 API Version of Cluster API (contract):

NAME                    NAMESPACE                           TYPE                     CURRENT VERSION   NEXT VERSION
bootstrap-kubeadm       capi-kubeadm-bootstrap-system       BootstrapProvider        v0.4.0           v1.0.0
control-plane-kubeadm   capi-kubeadm-control-plane-system   ControlPlaneProvider     v0.4.0           v1.0.0
cluster-api             capi-system                         CoreProvider             v0.4.0           v1.0.0
infrastructure-docker   capd-system                         InfrastructureProvider   v0.4.0           v1.0.0

You can now apply the upgrade by executing the following command:

   clusterctl upgrade apply --contract v1beta1

The output contains the latest release available for each Cluster API contract version. available at the moment.

upgrade apply

After choosing the desired option for the upgrade, you can run the following command to upgrade all the providers in the management cluster. This upgrades all the providers to the latest stable releases.

clusterctl upgrade apply --contract v1beta1

The upgrade process is composed by three steps:

• Check the cert-manager version, and if necessary, upgrade it.
• Delete the current version of the provider components, while preserving the namespace where the provider components are hosted and the provider’s CRDs.
• Install the new version of the provider components.

Please note that clusterctl does not upgrade Cluster API objects (Clusters, MachineDeployments, Machine etc.); upgrading such objects are the responsibility of the provider’s controllers.

It is also possible to explicitly upgrade one or more components to specific versions.

clusterctl upgrade apply \
    --core cluster-api:v1.2.4 \
    --infrastructure docker:v1.2.4

clusterctl delete

The clusterctl delete command deletes the provider components from the management cluster.

The operation is designed to prevent accidental deletion of user created objects. For example:

clusterctl delete --infrastructure aws

This command deletes the AWS infrastructure provider components, while preserving the namespace where the provider components are hosted and the provider’s CRDs.

If you want to delete all the providers in a single operation, you can use the --all flag.

clusterctl delete --all

clusterctl completion

The clusterctl completion command outputs shell completion code for the specified shell (bash or zsh). The shell code must be evaluated to provide interactive completion of clusterctl commands.

Bash

To install bash-completion on macOS, use Homebrew:

brew install bash-completion

Once installed, bash_completion must be evaluated. This can be done by adding the following line to the ~/.bash_profile.

[[ -r "$(brew --prefix)/etc/profile.d/bash_completion.sh" ]] && . "$(brew --prefix)/etc/profile.d/bash_completion.sh"

If bash-completion is not installed on Linux, please install the ‘bash-completion’ package via your distribution’s package manager.

You now have to ensure that the clusterctl completion script gets sourced in all your shell sessions. There are multiple ways to achieve this:

• Source the completion script in your ~/.bash_profile file:

  source <(clusterctl completion bash)
  

• Add the completion script to the /usr/local/etc/bash_completion.d directory:

  clusterctl completion bash >/usr/local/etc/bash_completion.d/clusterctl
  

Zsh

The clusterctl completion script for Zsh can be generated with the command clusterctl completion zsh.

If shell completion is not already enabled in your environment you will need to enable it. You can execute the following once:

echo "autoload -U compinit; compinit" >> ~/.zshrc

To load completions for each session, execute once:

clusterctl completion zsh > "${fpath[1]}/_clusterctl"

You will need to start a new shell for this setup to take effect.

clusterctl alpha rollout

The clusterctl alpha rollout command manages the rollout of a Cluster API resource. It consists of several sub-commands which are documented below.

Restart

Use the restart sub-command to force an immediate rollout. Note that rollout refers to the replacement of existing machines with new machines using the desired rollout strategy (default: rolling update). For example, here the MachineDeployment my-md-0 will be immediately rolled out:

clusterctl alpha rollout restart machinedeployment/my-md-0

Pause/Resume

Use the pause sub-command to pause a Cluster API resource. The command is a NOP if the resource is already paused. Note that internally, this command sets the Paused field within the resource spec (e.g. MachineDeployment.Spec.Paused) to true.

clusterctl alpha rollout pause machinedeployment/my-md-0

Use the resume sub-command to resume a currently paused Cluster API resource. The command is a NOP if the resource is currently not paused.

clusterctl alpha rollout resume machinedeployment/my-md-0

clusterctl config repositories

Display the list of providers and their repository configurations.

clusterctl ships with a list of known providers; if necessary, edit $XDG_CONFIG_HOME/cluster-api/clusterctl.yaml file to add a new provider or to customize existing ones.

clusterctl help

Help provides help for any command in the application. Simply type clusterctl help [command] for full details.

clusterctl version

Print clusterctl version.

clusterctl init list-images

Lists the container images required for initializing the management cluster.

clusterctl Configuration File

The clusterctl config file is located at $XDG_CONFIG_HOME/cluster-api/clusterctl.yaml. It can be used to:

• Customize the list of providers and provider repositories.
• Provide configuration values to be used for variable substitution when installing providers or creating clusters.
• Define image overrides for air-gapped environments.

Provider repositories

The clusterctl CLI is designed to work with providers implementing the clusterctl Provider Contract.

Each provider is expected to define a provider repository, a well-known place where release assets are published.

By default, clusterctl ships with providers sponsored by SIG Cluster Lifecycle. Use clusterctl config repositories to get a list of supported providers and their repository configuration.

Users can customize the list of available providers using the clusterctl configuration file, as shown in the following example:

providers:
  # add a custom provider
  - name: "my-infra-provider"
    url: "https://github.com/myorg/myrepo/releases/latest/infrastructure-components.yaml"
    type: "InfrastructureProvider"
  # override a pre-defined provider
  - name: "cluster-api"
    url: "https://github.com/myorg/myforkofclusterapi/releases/latest/core-components.yaml"
    type: "CoreProvider"
  # add a custom provider on a self-hosted GitLab (host should start with "gitlab.")
  - name: "my-other-infra-provider"
    url: "https://gitlab.example.com/api/v4/projects/myorg%2Fmyrepo/packages/generic/myrepo/v1.2.3/infrastructure-components.yaml"
    type: "InfrastructureProvider"
  # override a pre-defined provider on a self-hosted GitLab (host should start with "gitlab.")
  - name: "kubeadm"
    url: "https://gitlab.example.com/api/v4/projects/external-packages%2Fcluster-api/packages/generic/cluster-api/v1.1.3/bootstrap-components.yaml"
    type: "BootstrapProvider"

See provider contract for instructions about how to set up a provider repository.

Note: It is possible to use the ${HOME} and ${CLUSTERCTL_REPOSITORY_PATH} environment variables in url.

Variables

When installing a provider clusterctl reads a YAML file that is published in the provider repository. While executing this operation, clusterctl can substitute certain variables with the ones provided by the user.

The same mechanism also applies when clusterctl reads the cluster templates YAML published in the repository, e.g. when injecting the Kubernetes version to use, or the number of worker machines to create.

The user can provide values using OS environment variables, but it is also possible to add variables in the clusterctl config file:

# Values for environment variable substitution
AWS_B64ENCODED_CREDENTIALS: XXXXXXXX

The format of keys should always be UPPERCASE_WITH_UNDERSCORE for both OS environment variables and in the clusterctl config file (NOTE: this limitation derives from Viper, the library we are using internally to retrieve variables).

In case a variable is defined both in the config file and as an OS environment variable, the environment variable takes precedence.

Cert-Manager configuration

While doing init, clusterctl checks if there is a version of cert-manager already installed. If not, clusterctl will install a default version.

By default, cert-manager will be fetched from https://github.com/cert-manager/cert-manager/releases; however, if the user wants to use a different repository, it is possible to use the following configuration:

cert-manager:
  url: "/Users/foo/.config/cluster-api/dev-repository/cert-manager/latest/cert-manager.yaml"

Note: It is possible to use the ${HOME} and ${CLUSTERCTL_REPOSITORY_PATH} environment variables in url.

Similarly, it is possible to override the default version installed by clusterctl by configuring:

cert-manager:
  ...
  version: "v1.1.1"

For situations when resources are limited or the network is slow, the cert-manager wait time to be running can be customized by adding a field to the clusterctl config file, for example:

cert-manager:
  ...
  timeout: 15m

The value string is a possibly signed sequence of decimal numbers, each with optional fraction and a unit suffix, such as “300ms”, “-1.5h” or “2h45m”. Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h”.

If no value is specified, or the format is invalid, the default value of 10 minutes will be used.

Please note that the configuration above will be considered also when doing clusterctl upgrade plan or clusterctl upgrade apply.

For situations when you are using your own cert-manager installation, you can tell clusterctl init that cert-manager is already installed by adding a field to the clusterctl config file, for example:

cert-manager:
  ...
  externallyProvisioned: true
  timeout: 15m
  ...

If the externallyProvisioned flag is set, clusterctl will not install cert-manager but only test if it is proper working. If cert-manager is not working the test will be repeated until timeout expires; after that clusterctl int will stop and report an error.

Migrating to user-managed cert-manager

You may want to migrate to a user-managed cert-manager further down the line, after initialising cert-manager on the management cluster through clusterctl.

clusterctl looks for the label clusterctl.cluster.x-k8s.io/core=cert-manager on all api resources in the cert-manager namespace. If it finds the label, clusterctl will manage the cert-manager deployment. You can list all the resources with that label by running:

kubectl api-resources --verbs=list -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -A --selector=clusterctl.cluster.x-k8s.io/core=cert-manager

If you want to manage and install your own cert-manager, you’ll need to remove this label from all API resources.

Avoiding GitHub rate limiting

Follow this

Overrides Layer

clusterctl uses an overrides layer to read in injected provider components, cluster templates and metadata. By default, it reads the files from $XDG_CONFIG_HOME/cluster-api/overrides.

The directory structure under the overrides directory should follow the template:

<providerType-providerName>/<version>/<fileName>

For example,

├── bootstrap-kubeadm
│   └── v1.1.5
│       └── bootstrap-components.yaml
├── cluster-api
│   └── v1.1.5
│       └── core-components.yaml
├── control-plane-kubeadm
│   └── v1.1.5
│       └── control-plane-components.yaml
└── infrastructure-aws
    └── v0.5.0
            ├── cluster-template-dev.yaml
            └── infrastructure-components.yaml

For developers who want to generate the overrides layer, see Build artifacts locally.

Once these overrides are specified, clusterctl will use them instead of getting the values from the default or specified providers.

One example usage of the overrides layer is that it allows you to deploy clusters with custom templates that may not be available from the official provider repositories. For example, you can now do:

clusterctl generate cluster mycluster --flavor dev --infrastructure aws:v0.5.0 -v5

The -v5 provides verbose logging which will confirm the usage of the override file.

Using Override="cluster-template-dev.yaml" Provider="infrastructure-aws" Version="v0.5.0"

Another example, if you would like to deploy a custom version of CAPA, you can make changes to infrastructure-components.yaml in the overrides folder and run,

clusterctl init --infrastructure aws:v0.5.0 -v5

...
Using Override="infrastructure-components.yaml" Provider="infrastructure-aws" Version="v0.5.0"
...

If you prefer to have the overrides directory at a different location (e.g. /Users/foobar/workspace/dev-releases) you can specify the overrides directory in the clusterctl config file as

overridesFolder: /Users/foobar/workspace/dev-releases

Note: It is possible to use the ${HOME} and ${CLUSTERCTL_REPOSITORY_PATH} environment variables in overridesFolder.

Image overrides

When working in air-gapped environments, it’s necessary to alter the manifests to be installed in order to pull images from a local/custom image repository instead of public ones (e.g. gcr.io, or quay.io).

The clusterctl configuration file can be used to instruct clusterctl to override images automatically.

This can be achieved by adding an images configuration entry as shown in the example:

images:
  all:
    repository: myorg.io/local-repo

Please note that the image override feature allows for more fine-grained configuration, allowing to set image overrides for specific components, for example:

images:
  all:
    repository: myorg.io/local-repo
  cert-manager:
    tag: v1.5.3

In this example we are overriding the image repository for all the components and the image tag for all the images in the cert-manager component.

If required to alter only a specific image you can use:

images:
  all:
    repository: myorg.io/local-repo
  cert-manager/cert-manager-cainjector:
    tag: v1.5.3

Additionally, you can override the image name itself. This is useful when images have been mirrored with different names:

images:
  all:
    repository: myorg.io/local-repo
  cluster-api:
    name: mirrored-cluster-api-controller

This would transform registry.k8s.io/cluster-api/cluster-api-controller:v1.10.6 into myorg.io/local-repo/mirrored-cluster-api-controller:v1.10.6.

Or you can specify all three fields to completely override the image:

images:
  cluster-api:
    repository: myorg.io/local-repo
    name: mirrored-cluster-api-controller
    tag: v1.10.6

This would transform registry.k8s.io/cluster-api/cluster-api-controller:v1.8.0 into myorg.io/local-repo/mirrored-cluster-api-controller:v1.10.6, replacing both the image location and version.

Debugging/Logging

To have more verbose logs you can use the -v flag when running the clusterctl and set the level of the logging verbose with a positive integer number, ie. -v 3.

If you do not want to use the flag every time you issue a command you can set the environment variable CLUSTERCTL_LOG_LEVEL or set the variable in the clusterctl config file located by default at $XDG_CONFIG_HOME/cluster-api/clusterctl.yaml.

Skip checking for updates

clusterctl automatically checks for new versions every time it is used. If you do not want clusterctl to check for new updates you can set the environment variable CLUSTERCTL_DISABLE_VERSIONCHECK to "true" or set the variable in the clusterctl config file located by default at $XDG_CONFIG_HOME/cluster-api/clusterctl.yaml.

clusterctl for Developers

This document describes how to use clusterctl during the development workflow.

Prerequisites

• A Cluster API development setup (go, git, kind v0.9 or newer, Docker v19.03 or newer etc.)
• A local clone of the Cluster API GitHub repository
• A local clone of the GitHub repositories for the providers you want to install

Build clusterctl

From the root of the local copy of Cluster API, you can build the clusterctl binary by running:

make clusterctl

The output of the build is saved in the bin/ folder; In order to use it you have to specify the full path, create an alias or copy it into a folder under your $PATH.

Use local artifacts

Clusterctl by default uses artifacts published in the providers repositories; during the development workflow you may want to use artifacts from your local workstation.

There are two options to do so:

• Use the overrides layer, when you want to override a single published artifact with a local one.
• Create a local repository, when you want to avoid using published artifacts and use the local ones instead.

If you want to create a local artifact, follow these instructions:

Build artifacts locally

In order to build artifacts for the CAPI core provider, the kubeadm bootstrap provider, the kubeadm control plane provider and the Docker infrastructure provider:

make docker-build REGISTRY=gcr.io/k8s-staging-cluster-api PULL_POLICY=IfNotPresent

Create a clusterctl-settings.json file

Next, create a clusterctl-settings.json file and place it in your local copy of Cluster API. This file will be used by create-local-repository.py. Here is an example:

{
  "providers": ["cluster-api","bootstrap-kubeadm","control-plane-kubeadm", "infrastructure-aws", "infrastructure-docker"],
  "provider_repos": ["../cluster-api-provider-aws"]
}

providers (Array[]String, default=[]): A list of the providers to enable. See available providers for more details.

provider_repos (Array[]String, default=[]): A list of paths to all the providers you want to use. Each provider must have a clusterctl-settings.json file describing how to build the provider assets.

Create the local repository

Run the create-local-repository hack from the root of the local copy of Cluster API:

cmd/clusterctl/hack/create-local-repository.py

The script reads from the source folders for the providers you want to install, builds the providers’ assets, and places them in a local repository folder located under $XDG_CONFIG_HOME/cluster-api/dev-repository/. Additionally, the command output provides you the clusterctl init command with all the necessary flags. The output should be similar to:

clusterctl local overrides generated from local repositories for the cluster-api, bootstrap-kubeadm, control-plane-kubeadm, infrastructure-docker, infrastructure-aws providers.
in order to use them, please run:

clusterctl init \
   --core cluster-api:v0.3.8 \
   --bootstrap kubeadm:v0.3.8 \
   --control-plane kubeadm:v0.3.8 \
   --infrastructure aws:v0.5.0 \
   --infrastructure docker:v0.3.8 \
   --config $XDG_CONFIG_HOME/cluster-api/dev-repository/config.yaml

As you might notice, the command is using the $XDG_CONFIG_HOME/cluster-api/dev-repository/config.yaml config file, containing all the required setting to make clusterctl use the local repository (it fallbacks to $HOME if $XDG_CONFIG_HOME is not set on your machine).

Available providers

The following providers are currently defined in the script:

• cluster-api
• bootstrap-kubeadm
• control-plane-kubeadm
• infrastructure-docker

More providers can be added by editing the clusterctl-settings.json in your local copy of Cluster API; please note that each provider_repo should have its own clusterctl-settings.json describing how to build the provider assets, e.g.

{
  "name": "infrastructure-aws",
  "config": {
    "componentsFile": "infrastructure-components.yaml",
    "nextVersion": "v0.5.0"
  }
}

Create a kind management cluster

kind can provide a Kubernetes cluster to be used as a management cluster. See Install and/or configure a Kubernetes cluster for more information.

Before running clusterctl init, you must ensure all the required images are available in the kind cluster.

This is always the case for images published in some image repository like Docker Hub or gcr.io, but it can’t be the case for images built locally; in this case, you can use kind load to move the images built locally. e.g.

kind load docker-image gcr.io/k8s-staging-cluster-api/cluster-api-controller-amd64:dev
kind load docker-image gcr.io/k8s-staging-cluster-api/kubeadm-bootstrap-controller-amd64:dev
kind load docker-image gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev
kind load docker-image gcr.io/k8s-staging-cluster-api/capd-manager-amd64:dev

to make the controller images available for the kubelet in the management cluster.

When the kind cluster is ready and all the required images are in place, run the clusterctl init command generated by the create-local-repository.py script.

Optionally, you may want to check if the components are running properly. The exact components are dependent on which providers you have initialized. Below is an example output with the Docker provider being installed.

kubectl get deploy -A | grep "cap\|cert"

capd-system                         capd-controller-manager                         1/1     1            1           25m
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager       1/1     1            1           25m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager   1/1     1            1           25m
capi-system                         capi-controller-manager                         1/1     1            1           25m
cert-manager                        cert-manager                                    1/1     1            1           27m
cert-manager                        cert-manager-cainjector                         1/1     1            1           27m
cert-manager                        cert-manager-webhook                            1/1     1            1           27m

Additional Notes for the Docker Provider

Select the appropriate Kubernetes version

When selecting the --kubernetes-version, ensure that the kindest/node image is available.

For example, assuming that on docker hub there is no image for version vX.Y.Z, therefore creating a CAPD workload cluster with --kubernetes-version=vX.Y.Z will fail. See issue 3795 for more details.

Get the kubeconfig for the workload cluster when using Docker Desktop

For Docker Desktop on macOS, Linux or Windows use kind to retrieve the kubeconfig.

kind get kubeconfig --name capi-quickstart > capi-quickstart.kubeconfig

Docker Engine for Linux works with the default clusterctl approach.

clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

Fix kubeconfig when using Docker Desktop and clusterctl

When retrieving the kubeconfig using clusterctl with Docker Desktop on macOS or Windows or Docker Desktop (Docker Engine works fine) on Linux, you’ll need to take a few extra steps to get the kubeconfig for a workload cluster created with the Docker provider.

clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig

To fix the kubeconfig run:

# Point the kubeconfig to the exposed port of the load balancer, rather than the inaccessible container IP.
sed -i -e "s/server:.*/server: https:\/\/$(docker port capi-quickstart-lb 6443/tcp | sed "s/0.0.0.0/127.0.0.1/")/g" ./capi-quickstart.kubeconfig

clusterctl Extensions with Plugins

You can extend clusterctl with plugins, similar to kubectl. Please refer to the kubectl plugin documentation for more information, as clusterctl plugins are implemented in the same way, with the exception of plugin distribution.

Installing clusterctl plugins

To install a clusterctl plugin, place the plugin’s executable file in any location on your PATH.

Writing clusterctl plugins

No plugin installation or pre-loading is required. Plugin executables inherit the environment from the clusterctl binary. A plugin determines the command it implements based on its name. For example, a plugin named clusterctl-foo provides the clusterctl foo command. The plugin executable should be installed in your PATH.

Example plugin

#!/bin/bash

# optional argument handling
if [[ "$1" == "version" ]]
then
echo "1.0.0"
exit 0
fi

# optional argument handling
if [[ "$1" == "example-env-var" ]]
then
    echo "$EXAMPLE_ENV_VAR"
    exit 0
fi

echo "I am a plugin named clusterctl-foo"

Using a plugin

To use a plugin, make the plugin executable:

sudo chmod +x ./clusterctl-foo

and place it anywhere in your PATH:

sudo mv ./clusterctl-foo /usr/local/bin

You may now invoke your plugin as a clusterctl command:

clusterctl foo

I am a plugin named clusterctl-foo

All args and flags are passed as-is to the executable:

clusterctl foo version

1.0.0

All environment variables are also passed as-is to the executable:

export EXAMPLE_ENV_VAR=example-value
clusterctl foo example-env-var

example-value

EXAMPLE_ENV_VAR=another-example-value clusterctl foo example-env-var

another-example-value

Additionally, the first argument that is passed to a plugin will always be the full path to the location where it was invoked ($0 would equal /usr/local/bin/clusterctl-foo in the example above).

Naming a plugin

A plugin determines the command path it implements based on its filename. Each sub-command in the path is separated by a dash (-). For example, a plugin for the command clusterctl foo bar baz would have the filename clusterctl-foo-bar-baz.

Developer Guide

Pieces of Cluster API

Cluster API is made up of many components, all of which need to be running for correct operation. For example, if you wanted to use Cluster API with AWS, you’d need to install both the cluster-api manager and the aws manager.

Cluster API includes a built-in provisioner, Docker, that’s suitable for using for testing and development. This guide will walk you through getting that daemon, known as CAPD, up and running.

Other providers may have additional steps you need to follow to get up and running.

Prerequisites

Docker

Iterating on the cluster API involves repeatedly building Docker containers. You’ll need the docker daemon v19.03 or newer available.

On MacOS systems using Lima is a viable alternative to Docker Desktop.

A Cluster

You’ll likely want an existing cluster as your management cluster. The easiest way to do this is with kind v0.9 or newer, as explained in the quick start.

Make sure your cluster is set as the default for kubectl. If it’s not, you will need to modify subsequent kubectl commands below.

A container registry

If you’re using kind, you’ll need a way to push your images to a registry so they can be pulled. You can instead side-load all images, but the registry workflow is lower-friction.

Most users test with GCR, but you could also use something like Docker Hub. If you choose not to use GCR, you’ll need to set the REGISTRY environment variable.

Kustomize

You’ll need to install kustomize. There is a version of kustomize built into kubectl, but it does not have all the features of kustomize v3 and will not work.

Kubebuilder

You’ll need to install kubebuilder.

Envsubst

You’ll need envsubst to handle variable substitution in manifests.

The GNU gettext version of envsubst does not support default values, so you must use the drone/envsubst version.

go install github.com/drone/envsubst/v2/cmd/envsubst@latest

Cert-Manager

You’ll need to deploy cert-manager components on your management cluster, using kubectl

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.21.1/cert-manager.yaml

Ensure the cert-manager webhook service is ready before creating the Cluster API components.

This can be done by following instructions for manual verification from the cert-manager web site. Note: make sure to follow instructions for the release of cert-manager you are installing.

Development

Option 1: Tilt

Tilt is a tool for quickly building, pushing, and reloading Docker containers as part of a Kubernetes deployment. Many of the Cluster API engineers use it for quick iteration. Please see our Tilt instructions to get started.

Option 2: The Old-fashioned way

# Build all the images
make docker-build

# Push images
make docker-push

# Apply the manifests
kustomize build config/default | ~/go/bin/envsubst | kubectl apply -f -
kustomize build bootstrap/kubeadm/config/default | ~/go/bin/envsubst | kubectl apply -f -
kustomize build controlplane/kubeadm/config/default | ~/go/bin/envsubst | kubectl apply -f -
kustomize build test/infrastructure/docker/config/default | ~/go/bin/envsubst | kubectl apply -f -

Testing

Cluster API has a number of test suites available for you to run. Please visit the testing page for more information on each suite.

That’s it!

Now you can create CAPI objects! To test another iteration, you’ll need to follow the steps to build, push, update the manifests, and apply.

Videos explaining CAPI architecture and code walkthroughs

CAPI components and architecture

• Simplified Experience Of Building Cluster API Provider In Multitenant Cloud - October 2022
• Cluster API Intro and Deep Dive - May 2022 v1beta1
• Cluster API Deep Dive - Dec 2020 v1alpha3
• Cluster API Deep Dive - Sept 2020 v1alpha3
• Declarative Kubernetes Clusters with Cluster API - Oct 2020 v1alpha3
• TGI Kubernetes 178: ClusterAPI - ClusterClass & Managed Topologies - Dec 2020 v1beta1

Additional ClusterAPI KubeCon talks

• SIG Cluster Lifecycle Intro & Future - November 2023
• Cluster API Deep Dive: Improving Performance up to 2k Clusters - November 2023
• Leveraging Cluster-API for Production-Ready Multi-Regional Infrastructures - November 2023
• The Stars Look Very Different Today”: Kubernetes and Cloud Native at the SKA Observatory - November 2023
• 15,000 Minecraft Players Vs One K8s Cluster. Who Wins? - November 2023
• Cluster API Providers: Intro, Deep Dive, and Community! - April 2023
• Ephemeral Clusters as a Service with ClusterAPI and GitOps - April 2023
• The Power of Self-Managing Clusters - April 2023
• How to Turn Release Management from Duty to Fun - April 2023
• Tilt Your World! Lessons Learned in Improving Dev Productivity with Tilt - April 2023
• How Adobe Planned For Scale With Argo CD, Cluster API, And VCluster - October 2022
• Bare-Metal Chronicles: Intertwinement Of Tinkerbell, Cluster API And GitOps - October 2022
• Running Isolated VirtualClusters With Kata & Cluster API - October 2022
• SIG Cluster Lifecycle Intro - October 2022
• How to Migrate 700 Kubernetes Clusters to Cluster API with Zero Downtime - May 2022
• Build Your Own Cluster API Provider the Easy Way - May 2022

Tutorials

• kubectl Create Cluster: Production-ready Kubernetes with Cluster API 1.0 - October 2022

  Source code

• So You Want To Develop a Cluster API Provider? - October 2022

Code walkthroughs

• CAPD Deep Dive - March 2021 v1alpha4
• API conversion code walkthrough - January 2022

Let’s chat about …

We are currently hosting “Let’s chat about …” sessions where we are talking about topics relevant to contributors and users of the Cluster API project. For more details and an up-to-date list of recordings of past sessions please see Let’s chat about ….

• Local CAPI development and debugging with Tilt (EMEA/Americas) - February 2022
• Local CAPI development and debugging with Tilt (APAC/EMEA) - February 2022
• Code structure & Makefile targets (EMEA/Americas) - February 2022
• Code structure & Makefile targets (APAC/EMEA) - February 2022

CAPI e2e Deep Dive*

These are Deep-dive sessions with the CI team to investigate failing and flaking tests.

• CAPI e2e Deep Dive - 2023-05-11
• CAPI e2e Deep Dive - 2023-05-04

Developing “core” Cluster API

This section of the book is about developing “core” Cluster API.

With “core” Cluster API we refer to the common set of API and controllers that are required to run any Cluster API provider.

Please note that in the Cluster API code base, side by side of “core” Cluster API components there is also a limited number of in-tree providers:

• Kubeadm bootstrap provider (CAPBK)
• Kubeadm control plane provider (KCP)
• Docker infrastructure provider (CAPD) - The Docker provider is not designed for production use and is intended for development & test only.

Please refer to Developing providers for documentation about in-tree providers (and out of tree providers too).

Developing Cluster API with Tilt

Overview

This document describes how to use kind and Tilt for a simplified workflow that offers easy deployments and rapid iterative builds.

Prerequisites

1. Docker: v19.03 or newer (on MacOS e.g. via Lima)
2. kind: v0.32.0 or newer
3. Tilt: v0.33.18 or newer
4. kustomize: provided via make kustomize
5. helm: v3.7.1 or newer
6. Clone the Cluster API repository locally
7. Clone the provider(s) you want to deploy locally as well

Getting started

Create a kind cluster

This guide offers instructions for using the following CAPI infrastructure providers for running a development environment without using real machines or cloud resources:

• CAPD - uses Docker containers as workload cluster nodes
• CAPK - uses KubeVirt VMs as workload cluster nodes

CAPD is the default as it’s more lightweight and requires less setup. KubeVirt is useful when Docker isn’t suitable for whatever reason. Other infrastructure providers may be enabled as well (see below).

make kind-cluster

make kind-cluster-kubevirt

You can see the status of the cluster with:

kubectl cluster-info --context kind-capi-test

Create a tilt-settings file

Next, create a tilt-settings.yaml file and place it in your local copy of cluster-api.

Here are some examples:

default_registry: gcr.io/your-project-name-here
enable_providers:
- docker
- kubeadm-bootstrap
- kubeadm-control-plane

enable_providers:
- kubevirt
- kubeadm-bootstrap
- kubeadm-control-plane
provider_repos:
# Path to a local clone of CAPK (replace with actual path)
- ../cluster-api-provider-kubevirt
kustomize_substitutions:
  # CAPK needs access to the containerd socket (replace with actual path)
  CRI_PATH: "/var/run/containerd/containerd.sock"
  KUBERNETES_VERSION: "v1.30.1"
  # An example - replace with an appropriate container disk image for the desired k8s version
  NODE_VM_IMAGE_TEMPLATE: "quay.io/capk/ubuntu-2204-container-disk:v1.30.1"
# Allow deploying CAPK workload clusters from the Tilt UI (optional)
template_dirs:
  kubevirt:
  - ../cluster-api-provider-kubevirt/templates

Other infrastructure providers may be added to the cluster using local clones and a configuration similar to the following:

default_registry: gcr.io/your-project-name-here
provider_repos:
- ../cluster-api-provider-aws
enable_providers:
- aws
- kubeadm-bootstrap
- kubeadm-control-plane

tilt-settings fields

allowed_contexts (Array, default=[]): A list of kubeconfig contexts Tilt is allowed to use. See the Tilt documentation on allow_k8s_contexts for more details.

default_registry (String, default=[]): The image registry to use if you need to push images. See the Tilt documentation for more details. Please note that, in case you are not using a local registry, this value is required; additionally, the Cluster API Tiltfile protects you from accidental push on gcr.io/k8s-staging-cluster-api.

build_engine (String, default=“docker”): The engine used to build images. Can either be docker or podman. NB: the default is dynamic and will be “podman” if the string “Podman Engine” is found in docker version (or in podman version if the command fails).

kind_cluster_name (String, default=“capi-test”): The name of the kind cluster to use when preloading images.

provider_repos (Array[]String, default=[]): A list of paths to all the providers you want to use. Each provider must have a tilt-provider.yaml or tilt-provider.json file describing how to build the provider.

enable_providers (Array[]String, default=[‘docker’]): A list of the providers to enable. See available providers for more details.

enable_core_provider (bool, default=true): By default, the core provider is enabled. This allows to disable it.

preload_images (bool, default=true): By default, images are preloaded into the kind cluster. This works on most platforms but can fail on Apple Silicon or Docker v29+. Set this to false to skip preloading and let the kubelet pull images on demand.

template_dirs (Map{String: Array[]String}, default={“docker”: [ “./test/infrastructure/docker/templates”]}): A map of providers to directories containing cluster templates. An example of the field is given below. See Deploying a workload cluster for how this is used.

template_dirs:
  docker:
  - ./test/infrastructure/docker/templates
  - <other-template-dir>
  azure:
  - <azure-template-dir>
  aws:
  - <aws-template-dir>
  gcp:
  - <gcp-template-dir>

kustomize_substitutions (Map{String: String}, default={}): An optional map of substitutions for ${}-style placeholders in the provider’s yaml. These substitutions are also used when deploying cluster templates. See Deploying a workload cluster.

Note: When running E2E tests locally using an existing cluster managed by Tilt, the following substitutions are required for successful tests:

kustomize_substitutions:
  CLUSTER_TOPOLOGY: "true"
  EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION: "true"
  EXP_RUNTIME_SDK: "true"
  EXP_MACHINE_SET_PREFLIGHT_CHECKS: "true"

kustomize_substitutions:
  AWS_B64ENCODED_CREDENTIALS: "your credentials here"

AZURE_SUBSCRIPTION_ID=$(az account show --query id --output tsv)
az account set --subscription $AZURE_SUBSCRIPTION_ID

AZURE_SERVICE_PRINCIPAL_NAME=ServicePrincipalName

AZURE_TENANT_ID=$(az account show --query tenantId --output tsv)
AZURE_CLIENT_SECRET=$(az ad sp create-for-rbac --name http://$AZURE_SERVICE_PRINCIPAL_NAME --query password --output tsv)
AZURE_CLIENT_ID=$(az ad sp show --id http://$AZURE_SERVICE_PRINCIPAL_NAME --query appId --output tsv)

  cat <<EOF
  kustomize_substitutions:
     AZURE_SUBSCRIPTION_ID_B64: "$(echo "${AZURE_SUBSCRIPTION_ID}" | tr -d '\n' | base64 | tr -d '\n')"
     AZURE_TENANT_ID_B64: "$(echo "${AZURE_TENANT_ID}" | tr -d '\n' | base64 | tr -d '\n')"
     AZURE_CLIENT_SECRET_B64: "$(echo "${AZURE_CLIENT_SECRET}" | tr -d '\n' | base64 | tr -d '\n')"
     AZURE_CLIENT_ID_B64: "$(echo "${AZURE_CLIENT_ID}" | tr -d '\n' | base64 | tr -d '\n')"
  EOF

kustomize_substitutions:
  DO_B64ENCODED_CREDENTIALS: "your credentials here"

base64 -i ~/path/to/gcp/credentials.json

kustomize_substitutions:
  GCP_B64ENCODED_CREDENTIALS: "your credentials here"

kustomize_substitutions:
  VSPHERE_USERNAME: "administrator@vsphere.local"
  VSPHERE_PASSWORD: "Admin123"

deploy_observability ([string], default=[]): If set, installs on the dev cluster one of more observability tools. Important! This feature requires the helm command to be available in the user’s path.

Supported values are:

• grafana*: To create dashboards and query loki, prometheus and tempo.
• kube-state-metrics: For exposing metrics for Kubernetes and CAPI resources to prometheus.
• loki: To receive and store logs.
• metrics-server: To enable kubectl top node/pod.
• prometheus*: For collecting metrics from Kubernetes.
• alloy: For providing pod logs to loki.
• parca*: For visualizing profiling data.
• tempo: To store traces.
• visualizer*: Visualize Cluster API resources for each cluster, provide quick access to the specs and status of any resource.
• headlamp*: A Kubernetes web UI with the Cluster API plugin for browsing and managing CAPI resources.

*: Note: the UI will be accessible via a link in the tilt console

additional_kustomizations (map[string]string, default={}): If set, install the additional resources built using kustomize to the cluster. Example:

additional_kustomizations:
  capv-metrics: ../cluster-api-provider-vsphere/config/metrics

debug (Map{string: Map} default{}): A map of named configurations for the provider. The key is the name of the provider.

Supported settings:

• port (int, default=0 (disabled)): If set to anything other than 0, then Tilt will run the provider with delve and port forward the delve server to localhost on the specified debug port. This can then be used with IDEs such as Visual Studio Code, Goland and IntelliJ.

• continue (bool, default=true): By default, Tilt will run delve with --continue, such that any provider with debugging turned on will run normally unless specifically having a breakpoint entered. Change to false if you do not want the controller to start at all by default.

• profiler_port (int, default=0 (disabled)): If set to anything other than 0, then Tilt will enable the profiler with --profiler-address and set up a port forward. A “profiler” link will be visible in the Tilt Web UI for the controller.

• metrics_port (int, default=0 (disabled)): If set to anything other than 0, then Tilt will port forward to the default metrics port. A “metrics” link will be visible in the Tilt Web UI for the controller.

• race_detector (bool, default=false) (Linux amd64 only): If enabled, Tilt will compile the specified controller with cgo and statically compile in the system glibc and enable the race detector. Currently, this is only supported when building on Linux amd64 systems. You must install glibc-static or have libc.a available for this to work.

  Example: Using the configuration below:

    debug:
      core:
        continue: false
        port: 30000
        profiler_port: 40000
        metrics_port: 40001
  

  Wiring up debuggers

  Visual Studio

  When using the example above, the core CAPI controller can be debugged in Visual Studio Code using the following launch configuration:

  {
    "version": "0.2.0",
    "configurations": [
      {
        "name": "Core CAPI Controller",
        "type": "go",
        "request": "attach",
        "mode": "remote",
        "remotePath": "",
        "port": 30000,
        "host": "127.0.0.1",
        "showLog": true,
        "trace": "log",
        "logOutput": "rpc"
      }
    ]
  }
  

  Goland / IntelliJ

  With the above example, you can configure a Go Remote run/debug configuration pointing at port 30000.

deploy_cert_manager (Boolean, default=true): Deploys cert-manager into the cluster for use for webhook registration.

trigger_mode (String, default=auto): Optional setting to configure if tilt should automatically rebuild on changes. Set to manual to disable auto-rebuilding and require users to trigger rebuilds of individual changed components through the UI.

extra_args (Object, default={}): A mapping of provider to additional arguments to pass to the main binary configured for this provider. Each item in the array will be passed in to the manager for the given provider.

Example:

extra_args:
  kubeadm-bootstrap:
  - --logging-format=json

With this config, the respective managers will be invoked with:

manager --logging-format=json

Create a kind cluster and run Tilt!

To create a pre-configured kind cluster (if you have not already done so) and launch your development environment, run

make tilt-up

This will open the command-line HUD as well as a web browser interface. You can monitor Tilt’s status in either location. After a brief amount of time, you should have a running development environment, and you should now be able to create a cluster. There are example worker cluster configs available. These can be customized for your specific needs.

Deploying a workload cluster

After your kind management cluster is up and running with Tilt, you can deploy a workload clusters in the Tilt web UI based off of YAML templates from the directories specified in the template_dirs field from the tilt-settings.yaml file (default ./test/infrastructure/docker/templates).

Templates should be named according to clusterctl conventions:

• template files must be named cluster-template-{name}.yaml; those files will be accessible in the Tilt web UI under the label grouping {provider-label}.templates, i.e. CAPD.templates.
• cluster class files must be named clusterclass-{name}.yaml; those file will be accessible in the Tilt web UI under the label grouping {provider-label}.clusterclasses, i.e. CAPD.clusterclasses.

By selecting one of those items in the Tilt web UI set of buttons will appear, allowing to create - with a dropdown for customizing variable substitutions - or delete clusters. Custom values for variable substitutions can be set using kustomize_substitutions in tilt-settings.yaml, e.g.

kustomize_substitutions:
  NAMESPACE: "default"
  KUBERNETES_VERSION: "v1.36.1"
  CONTROL_PLANE_MACHINE_COUNT: "1"
  WORKER_MACHINE_COUNT: "3"
# Note: kustomize substitutions expects the values to be strings. This can be achieved by wrapping the values in quotation marks.

Cleaning up your kind cluster and development environment

After stopping Tilt, you can clean up your kind cluster and development environment by running

make clean-kind

To remove all generated files, run

make clean

Note that you must run make clean or make clean-charts to fetch new versions of charts deployed using deploy_observability in tilt-settings.yaml.

Use of clusterctl

When the worker cluster has been created using tilt, clusterctl should not be used for management operations; this is because tilt doesn’t initialize providers on the management cluster like clusterctl init does, so some of the clusterctl commands like clusterctl config won’t work.

This limitation is an acceptable trade-off while executing fast dev-test iterations on controllers logic. If instead you are interested in testing clusterctl workflows, you should refer to the clusterctl developer instructions.

Available providers

The following providers are currently defined in the Tiltfile:

• core: cluster-api itself
• kubeadm-bootstrap: kubeadm bootstrap provider
• kubeadm-control-plane: kubeadm control-plane provider
• docker: Docker infrastructure provider
• in-memory: In-memory infrastructure provider
• test-extension: Runtime extension used by CAPI E2E tests

Additional providers can be added by following the procedure described in following paragraphs:

tilt-provider configuration

A provider must supply a tilt-provider.yaml file describing how to build it. Here is an example:

name: aws
config:
  image: "gcr.io/k8s-staging-cluster-api-aws/cluster-api-aws-controller"
  live_reload_deps: ["main.go", "go.mod", "go.sum", "api", "cmd", "controllers", "pkg"]
  label: CAPA

config fields

image: the image for this provider, as referenced in the kustomize files. This must match; otherwise, Tilt won’t build it.

live_reload_deps: a list of files/directories to watch. If any of them changes, Tilt rebuilds the manager binary for the provider and performs a live update of the running container.

version: allows to define the version to be used for the Provider CR. If empty, a default version will be used.

additional_docker_helper_commands (String, default=“”): Additional commands to be run in the helper image docker build. e.g.

RUN wget -qO- https://dl.k8s.io/v1.21.2/kubernetes-client-linux-amd64.tar.gz | tar xvz
RUN wget -qO- https://get.docker.com | sh

additional_docker_build_commands (String, default=“”): Additional commands to be appended to the dockerfile. The manager image will use docker-slim, so to download files, use additional_helper_image_commands. e.g.

COPY --from=tilt-helper /usr/bin/docker /usr/bin/docker
COPY --from=tilt-helper /go/kubernetes/client/bin/kubectl /usr/bin/kubectl

kustomize_folder (String, default=config/default): The folder where the kustomize file for a provider is defined; the path is relative to the provider root folder.

kustomize_options ([]String, default=[]): Options to be applied when running kustomize for generating the yaml manifest for a provider. e.g. "kustomize_options": [ "--load-restrictor=LoadRestrictionsNone" ]

apply_provider_yaml (Bool, default=true): Whether to apply the provider yaml. Set to false if your provider does not have a ./config folder or you do not want it to be applied in the cluster.

go_main (String, default=“main.go”): The go main file if not located at the root of the folder

label (String, default=provider name): The label to be used to group provider components in the tilt UI in tilt version >= v0.22.2 (see https://blog.tilt.dev/2021/08/09/resource-grouping.html); as a convention, provider abbreviation should be used (CAPD, KCP etc.).

additional_resources ([]string, default=[]): A list of paths to yaml file to be loaded into the tilt cluster; e.g. use this to deploy an ExtensionConfig object for a RuntimeExtension provider.

additional_uncategorized_resources ([]string, default=[]): A list of paths to yaml file to be loaded into the tilt cluster; e.g. use this to deploy CustomResourceDefinitions. The difference compared to additional_resources is that it is deployed as part of uncategorized and accordingly not re-created together with providers.

resource_deps ([]string, default=[]): A list of tilt resource names to be installed before the current provider; e.g. set this to [“capi_controller”] to ensure that this provider gets installed after Cluster API.

Customizing Tilt

If you need to customize Tilt’s behavior, you can create files in cluster-api’s tilt.d directory. This file is ignored by git so you can be assured that any files you place here will never be checked in to source control.

These files are included after the providers map has been defined and after all the helper function definitions. This is immediately before the “real work” happens.

Under the covers, a.k.a “the real work”

At a high level, the Tiltfile performs the following actions:

1. Read tilt-settings.yaml
2. Configure the allowed Kubernetes contexts
3. Set the default registry
4. Define the providers map
5. Include user-defined Tilt files
6. Deploy cert-manager
7. Enable providers (core + what is listed in tilt-settings.yaml)
   1. Build the manager binary locally as a local_resource
   2. Invoke docker_build for the provider
   3. Invoke kustomize for the provider’s config/ directory

Live updates

Each provider in the providers map has a live_reload_deps list. This defines the files and/or directories that Tilt should monitor for changes. When a dependency is modified, Tilt rebuilds the provider’s manager binary on your local machine, copies the binary to the running container, and executes a restart script. This is significantly faster than rebuilding the container image for each change. It also helps keep the size of each development image as small as possible (the container images do not need the entire go toolchain, source code, module dependencies, etc.).

IDE support for Tiltfile

For IntelliJ, Syntax highlighting for the Tiltfile can be configured with a TextMate Bundle. For instructions, please see: Tiltfile TextMate Bundle.

For VSCode the Bazel plugin can be used, it provides syntax highlighting and auto-formatting. To enable it for Tiltfile a file association has to be configured via user settings:

"files.associations": {
  "Tiltfile": "starlark",
},

Using Podman

Podman can be used instead of Docker by following these actions:

1. Enable the podman unix socket:
   • on Linux/systemd: systemctl --user enable --now podman.socket
   • on macOS: create a podman machine with podman machine init
2. Set build_engine to podman in tilt-settings.yaml (optional, only if both Docker & podman are installed)
3. Define the env variable DOCKER_HOST to the right socket:
   • on Linux/systemd: export DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock
   • on macOS: export DOCKER_HOST=$(podman machine inspect <machine> | jq -r '.[0].ConnectionInfo.PodmanSocket.Path') where <machine> is the podman machine name
4. Run tilt up

NB: The socket defined by DOCKER_HOST is used only for the hack/tools/internal/tilt-prepare command, the image build is running the podman build/podman push commands.

Using Lima

Lima can be used instead of Docker Desktop. Please note that especially with CAPD the rootless template of Lima does not work.

The following command creates a working Lima machine for developing Cluster API with CAPD:

limactl start template://docker-rootful --name "docker" --tty=false \
  --set '.provision += {"mode":"system","script":"#!/bin/bash\nset -eux -o pipefail\ncat << EOF > \"/etc/sysctl.d/99-capi.conf\"\nfs.inotify.max_user_watches = 1048576\nfs.inotify.max_user_instances = 8192\nEOF\nsysctl -p \"/etc/sysctl.d/99-capi.conf\""}' \
  --set '.mounts[0] = {"location": "~", "writable": true}' \
  --memory 12 --cpus 10 --disk 64 \
  --vm-type vz --rosetta=true

After creating the Lima machine we need to set DOCKER_HOST to the correct path:

export DOCKER_HOST=$(limactl list "docker" --format 'unix://{{.Dir}}/sock/docker.sock')

Troubleshooting Tilt

Tilt is stuck

Sometimes tilt looks stuck when it’s waiting on connections.

Ensure that docker/podman is up and running and your kubernetes cluster is reachable.

Errors running tilt-prepare

failed to get current context from the KubeConfig file

• Ensure the cluster in the default context is reachable by running kubectl cluster-info
• Switch to the right context with kubectl config use-context
• Ensure the context is allowed, see allowed_contexts field

Cannot connect to the Docker daemon

• Ensure the docker daemon is running ;) or for podman see Using Podman
• If a DOCKER_HOST is specified:
  • check that the DOCKER_HOST has the correct prefix (usually unix://)
  • ensure docker/podman is listening on $DOCKER_HOST using fuser / lsof / netstat -u

Errors pulling/pushing to the registry

connection refused / denied / not found

Ensure the default_registry field is a valid registry where you can pull and push images.

server gave HTTP response to HTTPS client

By default all registries except localhost:5000 are accessed via HTTPS.

If you run a HTTP registry you may have to configure the registry in docker/podman.

For example, in podman a localhost:5001 registry configuration should be declared in /etc/containers/registries.conf.d with this content:

[[registry]]
location = "localhost:5001"
insecure = true

NB: on macOS this configuration should be done in the podman machine by running podman machine ssh <machine>.

Errors loading images in kind

You may try manually to load images in kind by running:

kind load docker-image --name=<kind_cluster> <image>

image: "..." not present locally

If you are running podman, you may have hit this bug: https://github.com/kubernetes-sigs/kind/issues/2760

The workaround is to create a docker symlink to your podman executable and try to load the images again.

Repository Layout

This page covers the repository structure and details about the directories in Cluster API.

cluster-api
└───.github
└───api
└───bootstrap
└───cmd
│   │   clusterctl
└───config
└───controllers
└───controlplane
└───dev
└───docs
└───errors
└───exp
└───feature
└───hack
└───internal
└───logos
└───scripts
└───test
└───util
└───version
└───webhooks
└───main.go
└───Makefile

GitHub

~/.github

Contains GitHub workflow configuration and templates for Pull requests, bug reports etc.

API

~/api

This folder is used to store types and their related resources present in CAPI core. It includes things like API types, spec/status definitions, condition types, simple webhook implementation, autogenerated, deepcopy and conversion files. Some examples of Cluster API types defined in this package include Cluster, ClusterClass, Machine, MachineSet, MachineDeployment and MachineHealthCheck.

API folder has subfolders for each supported API version.

Bootstrap

~/bootstrap

This folder contains Cluster API bootstrap provider Kubeadm (CABPK) which is a reference implementation of a Cluster API bootstrap provider. This folder contains the types and controllers responsible for generating a cloud-init or ignition configuration to turn a Machine into a Kubernetes Node. It is built and deployed as an independent provider alongside the Cluster API controller manager.

ControlPlane

~/controlplane

This folder contains a reference implementation of a Cluster API Control Plane provider - KubeadmControlPlane. This package contains the API types and controllers required to instantiate and manage a Kubernetes control plane. It is built and deployed as an independent provider alongside the Cluster API controller manager.

Cluster API Provider Docker

~/test/infrastructure/docker

This folder contains a reference implementation of an infrastructure provider for the Cluster API project using Docker. This provider is intended for development purposes only.

Clusterctl CLI

~/cmd/clusterctl

This folder contains Clusterctl, a CLI that can be used to deploy Cluster API and providers, generate cluster manifests, read the status of a cluster, and much more.

Manifest Generation

~/config

This is a Kubernetes manifest folder containing application resource configuration as kustomize YAML definitions. These are generated from other folders in the repo using make generate-manifests

Some of the subfolders are:

• ~/config/certmanager - It contains manifests like self-signed issuer CR and certificate CR useful for cert manager.

• ~/config/crd - It contains CRDs generated from types defined in api folder

• ~/config/manager - It contains manifest for the deployment of core Cluster API manager.

• ~/config/rbac - Manifests for RBAC resources generated from kubebuilder markers defined in controllers.

• ~/config/webhook - Manifest for webhooks generated from the markers defined in the web hook implementations present in api folder.

Note: Additional config containing manifests can be found in the packages for KubeadmControlPlane, KubeadmBootstrap and Cluster API Provider Docker.

Controllers

~/internal

This folder contains resources which are not meant to be used directly by users of Cluster API e.g. the implementation of controllers is present in ~/internal/controllers directory so that we can make changes in controller implementation without breaking users. This allows us to keep our api surface smaller and move faster.

~/controllers

This folder contains reconciler types which provide access to CAPI controllers present in ~/internal/controllers directory to our users. These types can be used by users to run any of the Cluster API controllers in an external program.

Documentation

~/docs

This folder is a place for proposals, developer release guidelines and the Cluster API book.

~/logos

Cluster API related logos and artwork

Tools

~/hack

This folder has scripts used for building, testing and developer workflow.

~/scripts

This folder consists of CI scripts related to setup, build and e2e tests. These are mostly called by CI jobs.

~/dev

This folder has example configuration for integrating Cluster API development with tools like IDEs.

Util, Feature and Errors

~/util

This folder contains utilities which are used across multiple CAPI package. These utils are also widely imported in provider implementations and by other users of CAPI.

~/feature

This package provides feature gate management used in Cluster API as well as providers. This implementation of feature gates is shared across all providers.

~/errors

This is a place for defining errors returned by CAPI. Error types defined here can be used by users of CAPI and the providers.

Experimental features

~/exp

This folder contains experimental features of CAPI. Experimental features are unreliable until they are promoted to the main repository. Each experimental feature is supposed to be present in a subfolder of ~/exp folder e.g. ClusterResourceSet is present inside ~/exp/addons folder. Historically, machine pool resources are not present in a sub-directory. Migrating them to a subfolder like ~/exp/machinepools is still pending as it can potentially break existing users who are relying on existing folder structure.

CRDs for experimental features are present outside ~/exp directory in ~/config folder. Also, these CRDs are deployed in the cluster irrespective of the feature gate value. These features can be enabled and disabled using feature gates supplied to the core Cluster API controller.

Webhooks

The api folder contains webhooks consisting of validators and defaults for many of the types in Cluster API.

~/internal/webhooks

This directory contains the implementation of some of the Cluster API webhooks. The internal implementation means that the methods supplied by this package cannot be imported by external code bases.

~/webhooks

This folder exposes the custom webhooks present in ~internal/webhooks to the users of CAPI.

Note: Additional webhook implementations can be found in the API packages for KubeadmControlPlane, KubeadmBootstrap and Cluster API Provider Docker.

Controllers

This section of the book provides an overview about “core” controllers in Cluster API.

Cluster Controller

The Cluster controller is responsible for reconciling the Cluster resource.

In order to allow Cluster provisioning on different type of infrastructure, The Cluster resource references an InfraCluster object, e.g. AWSCluster, GCPCluster etc.

The InfraCluster resource contract defines a set of rules a provider is expected to comply with in order to allow the expected interactions with the Cluster controller.

Among those rules:

• InfraCluster SHOULD report a controlplane endpoint for the Cluster
• InfraCluster SHOULD report available failure domains where machines should be placed in
• InfraCluster MUST report when Cluster’s infrastructure is fully provisioned
• InfraCluster SHOULD report conditions
• InfraCluster SHOULD report terminal failures

Similarly, in order to support different solutions for control plane management, The Cluster resource references an ControlPlane object, e.g. KubeadmControlPlane, EKSControlPlane etc.

Among those rules:

• ControlPlane SHOULD report a controlplane endpoint for the Cluster
• ControlPlane MUST report when Cluster’s control plane is fully provisioned
• ControlPlane MUST manage a KubeConfig secret
• ControlPlane SHOULD report conditions
• ControlPlane SHOULD report terminal failures

Considering all the info above, the Cluster controller’s main responsibilities are:

• Setting an OwnerReference on the infrastructure object referenced in Cluster.spec.infrastructureRef.
• Setting an OwnerReference on the control plane object referenced in Cluster.spec.controlPlaneRef.
• Keeping the Cluster’s status in sync with the InfraCluster and ControlPlane’s status.
• If no ControlPlane object is referenced, create a kubeconfig secret for workload clusters.
• Cleanup of all owned objects so that nothing is dangling after deletion.

Kubeconfig Secrets

In order to create a kubeconfig secret, it is required to have a certificate authority (CA) for the cluster.

If you are using the kubeadm bootstrap provider you do not have to provide any Cluster API secrets. It will generate all necessary CAs for you.

As alternative users can provide custom CA as described in Using Custom Certificates.

Last option, is to entirely bypass Cluster API kubeconfig generation by providing a kubeconfig secret formatted as described below.

Notes:

• Also renewal of the above certificate should be taken care out of band.
• This option does not prevent from providing a cluster CA which is required also for other purposes.

ClusterTopology Controller

The ClusterTopology controller reconciles the managed topology of a Cluster, as shown in the following diagram.

Its main responsibilities are to:

1. Reconcile Clusters based on templates defined in a ClusterClass and managed topology.
2. Create, update, delete managed topologies by continuously reconciling the topology managed resources.
3. Reconcile Cluster-specific customizations of a ClusterClass

The high level workflow of ClusterTopology reconciliation is shown below.

Additional information

• See ClusterClass proposal

ClusterResourceSet Controller

The ClusterResourceSet provides a mechanism for applying resources - e.g. pods, deployments, daemonsets, secrets, configMaps - to a cluster once it is created.

Its main responsibility is to automatically apply a set of resources to newly-created and existing Clusters. Resources will be applied only once.

Additional information

• See ClusterResourceSet proposal

MachineDeployment

A MachineDeployment orchestrates deployments over a fleet of MachineSets.

Its main responsibilities are:

• Adopting matching MachineSets not assigned to a MachineDeployment
• Adopting matching MachineSets not assigned to a Cluster
• Managing the Machine deployment process
  • Scaling up new MachineSets when changes are made
  • Scaling down old MachineSets when newer MachineSets replace them
• Updating the status of MachineDeployment objects

In-place propagation

Changes to the following fields of the MachineDeployment are propagated in-place to the MachineSet and do not trigger a full rollout:

• .annotations
• .spec.deletion.order
• .spec.template.metadata.labels
• .spec.template.metadata.annotations
• .spec.template.spec.minReadySeconds
• .spec.template.spec.deletion.nodeDrainTimeout
• .spec.template.spec.deletion.nodeDeletionTimeout
• .spec.template.spec.deletion.nodeVolumeDetachTimeout

Note: In cases where changes to any of these fields are paired with rollout causing changes, the new values are propagated only to the new MachineSet.

MachineSet

A MachineSet is an abstraction over Machines.

Its main responsibilities are:

• Adopting unowned Machines that aren’t assigned to a MachineSet
• Adopting unmanaged Machines that aren’t assigned a Cluster
• Booting a group of N machines
  • Monitoring the status of those booted machines

In-place propagation

Changes to the following fields of MachineSet are propagated in-place to the Machine without needing a full rollout:

• .spec.template.metadata.labels
• .spec.template.metadata.annotations
• .spec.template.spec.nodeDrainTimeout
• .spec.template.spec.nodeDeletionTimeout
• .spec.template.spec.nodeVolumeDetachTimeout

Changes to the following fields of MachineSet are propagated in-place to the InfrastructureMachine and BootstrapConfig:

• .spec.template.metadata.labels
• .spec.template.metadata.annotations

Note: Changes to these fields will not be propagated to Machines that are marked for deletion (example: because of scale down).

Machine Controller

The Machine controller is responsible for reconciling the Machine resource.

In order to allow Machine provisioning on different type of infrastructure, The Machine resource references an InfraMachine object, e.g. AWSMachine, GCMachine etc.

The InfraMachine resource contract defines a set of rules a provider is expected to comply with in order to allow the expected interactions with the Machine controller.

Among those rules:

• InfraMachine MUST report a provider ID for the Machine
• InfraMachine SHOULD take into account the failure domain where machines should be placed in
• InfraMachine SHOULD surface machine’s addresses to help operators when troubleshooting issues
• InfraMachine MUST report when Machine’s infrastructure is fully provisioned
• InfraMachine SHOULD report conditions
• InfraMachine SHOULD report terminal failures

Similarly, in order to support different machine bootstrappers, The Machine resource references a BootstrapConfig object, e.g. KubeadmBootstrapConfig etc.

The BootstrapConfig resource contract defines a set of rules a provider is expected to comply with in order to allow the expected interactions with the Machine controller.

Among those rules:

• BootstrapConfig MUST create a bootstrap data secret where machines should be placed in
• BootstrapConfig MUST report when Machine’s bootstrap data secret is fully provisioned
• BootstrapConfig SHOULD report conditions
• BootstrapConfig SHOULD report terminal failures
• BootstrapConfig SHOULD report taint Nodes at creation
• BootstrapConfig SHOULD create a sentinel file on machines

Considering all the info above, the Machine controller’s main responsibilities are:

• Setting an OwnerReference on the infrastructure object referenced in Machine.spec.infrastructureRef.
• Setting an OwnerReference on the bootstrap object referenced in Machine.spec.bootstrap.configRef.
• Keeping the Machine’s status in sync with the InfraMachine and BootstrapConfig’s status.
  • Finding Kubernetes nodes matching the expected providerID in the workload cluster.
  • Setting NodeRefs to be able to associate machines and Kubernetes nodes.
  • Monitor Kubernetes nodes and propagate labels to them.
• Cleanup of all owned objects so that nothing is dangling after deletion.
  • Drain nodes and wait for volumes being detached by CSI plugins.

After the machine controller sets the OwnerReferences on the associated objects, it waits for the bootstrap and infrastructure objects referenced by the machine to have the Status.initialization.dataSecretCreated field set to true. When the infrastructure object reports Status.initialization.provisioned, the machine controller will attempt to read its Spec.ProviderID and copy it into Machine.Spec.ProviderID.

The machine controller uses the kubeconfig for the new workload cluster to watch new nodes coming up. When a node appears with Node.Spec.ProviderID matching Machine.Spec.ProviderID, the machine controller transitions the associated machine into the Running state.

The following schema goes through machine phases and interactions with InfraMachine and BootstrapConfig happening at each step.

MachinePool Controller

📖 For conceptual information about MachinePools, when to use them, and how they compare to MachineDeployments, see the MachinePool Guide.

The MachinePool controller’s main responsibilities are:

• Setting an OwnerReference on each MachinePool object to:
  • The associated Cluster object.
  • The associated BootstrapConfig object.
  • The associated InfrastructureMachinePool object.
• Copy data from BootstrapConfig.Status.DataSecretName to MachinePool.Spec.Template.Spec.Bootstrap.DataSecretName if MachinePool.Spec.Template.Spec.Bootstrap.DataSecretName is empty.
• Setting NodeRefs on MachinePool instances to be able to associate them with Kubernetes nodes.
• Deleting Nodes in the target cluster when the associated MachinePool instance is deleted.
• Keeping the MachinePool’s Status object up to date with the InfrastructureMachinePool’s Status object.
• Finding Kubernetes nodes matching the expected providerIDs in the workload cluster.

After the machine pool controller sets the OwnerReferences on the associated objects, it waits for the bootstrap and infrastructure objects referenced by the machine to have the Status.Ready field set to true. When the infrastructure object is ready, the machine pool controller will attempt to read its Spec.ProviderIDList and copy it into MachinePool.Spec.ProviderIDList.

The machine pool controller uses the kubeconfig for the new workload cluster to watch new nodes coming up. When a node appears with a Node.Spec.ProviderID in MachinePool.Spec.ProviderIDList, the machine pool controller increments the number of ready replicas. When all replicas are ready and the infrastructure ref is also Ready, the machine pool controller marks the machine pool as Running.

Contracts

Cluster API

Cluster associations are made via labels.

Expected labels

Bootstrap provider

The BootstrapConfig object must have a status object.

The CRD name must have the format produced by sigs.k8s.io/cluster-api/util/contract.CalculateCRDName(Group, Kind).

To override the bootstrap provider, a user (or external system) can directly set the MachinePool.Spec.Bootstrap.DataSecretName field. This will mark the machine as ready for bootstrapping and no bootstrap data secret name will be copied from the BootstrapConfig object.

Required status fields

The status object must have several fields defined:

• ready - a boolean field indicating the bootstrap config data is generated and ready for use.
• dataSecretName - a string field referencing the name of the secret that stores the generated bootstrap data.

Optional status fields

The status object may define several fields that do not affect functionality if missing:

• failureReason - a string field explaining why a fatal error has occurred, if possible.
• failureMessage - a string field that holds the message contained by the error.

Note: once any of failureReason or failureMessage surface on the machine pool who is referencing the bootstrap config object, they cannot be restored anymore (it is considered a terminal error; the only way to recover is to delete and recreate the machine pool).

Example:

kind: MyBootstrapProviderConfig
apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
status:
    ready: true
    dataSecretName: "MyBootstrapSecret"

Infrastructure provider

The InfrastructureMachinePool object must have both spec and status objects.

The CRD name must have the format produced by sigs.k8s.io/cluster-api/util/contract.CalculateCRDName(Group, Kind).

Required spec fields

The spec object must have at least one field defined:

• providerIDList - the list of cloud provider IDs identifying the instances.

Required status fields

The status object must have at least one field defined:

• ready - a boolean field indicating if the infrastructure is ready to be used or not.

Optional status fields

The status object may define several fields that do not affect functionality if missing:

• failureReason - is a string that explains why a fatal error has occurred, if possible.
• failureMessage - is a string that holds the message contained by the error.
• infrastructureMachineKind - the kind of the InfraMachines. This should be set if the InfrastructureMachinePool plans to support MachinePool Machines.

Note: once any of failureReason or failureMessage surface on the machine pool who is referencing the InfrastructureMachinePool object, they cannot be restored anymore (it is considered a terminal error; the only way to recover is to delete and recreate the machine pool).

Note: Infrastructure providers can support MachinePool Machines by having the InfraMachinePool set the infrastructureMachineKind to the kind of their InfrastructureMachines. The InfrastructureMachinePool will be responsible for creating InfrastructureMachines as the MachinePool is scaled up, and the MachinePool controller will create Machines for each InfrastructureMachine and set the ownerRef. The InfrastructureMachinePool will be responsible for deleting the Machines as the MachinePool is scaled down in order for the Machine deletion workflow to function properly. In addition, the InfrastructureMachines must also have the following labels set by the InfrastructureMachinePool: cluster.x-k8s.io/cluster-name and cluster.x-k8s.io/pool-name. The MachinePoolNameLabel must also be formatted with capilabels.MustFormatValue() so that it will not exceed character limits.

Example

kind: MyMachinePool
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
spec:
    providerIDList:
      - cloud:////my-cloud-provider-id-0
      - cloud:////my-cloud-provider-id-1
status:
    ready: true
    infrastructureMachineKind: InfrastructureMachine

Externally Managed Autoscaler

A provider may implement an InfrastructureMachinePool that is externally managed by an autoscaler. For example, if you are using a Managed Kubernetes provider, it may include its own autoscaler solution. To indicate this to Cluster API, you would decorate the MachinePool object with the following annotation:

"cluster.x-k8s.io/replicas-managed-by": ""

Cluster API treats the annotation as a “boolean”, meaning that the presence of the annotation is sufficient to indicate external replica count management, with one exception: if the value is "false", then that indicates to Cluster API that replica enforcement is nominal, and managed by Cluster API.

Providers may choose to implement the cluster.x-k8s.io/replicas-managed-by annotation with different values (e.g., external-autoscaler, or karpenter) that may inform different provider-specific behaviors, but those values will have no effect upon Cluster API.

The effect upon Cluster API of this annotation is that during autoscaling events (initiated externally, not by Cluster API), when more or fewer MachinePool replicas are observed compared to the Spec.Replicas configuration, it will update its Status.Phase property to the value of "Scaling".

Example:

kind: MyMachinePool
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
spec:
    providerIDList:
      - cloud:////my-cloud-provider-id-0
      - cloud:////my-cloud-provider-id-1
      - cloud:////my-cloud-provider-id-2
    replicas: 1
status:
    ready: true
    phase: Scaling
    infrastructureMachineKind: InfrastructureMachine

It is the provider’s responsibility to update Cluster API’s Spec.Replicas property to the value observed in the underlying infra environment as it changes in response to external autoscaling behaviors. Once that is done, and the number of providerID items is equal to the Spec.Replicas property, the MachinePools’s Status.Phase property will be set to Running by Cluster API.

Secrets

The machine pool controller will use a secret in the following format:

MachineHealthCheck

A MachineHealthCheck is responsible for remediating unhealthy Machines.

Its main responsibilities are:

• Checking the health of Nodes in the workload clusters against a list of unhealthy conditions
• Remediating Machine’s for Nodes determined to be unhealthy

Logging

The Cluster API project is committed to improving the SRE/developer experience when troubleshooting issues, and logging plays an important part in this goal.

In Cluster API we strive to follow three principles while implementing logging:

• Logs are for SRE & developers, not for end users! Whenever an end user is required to read logs to understand what is happening in the system, most probably there is an opportunity for improvement of other observability in our API, like e.g. conditions and events.
• Navigating logs should be easy: We should make sure that SREs/Developers can easily drill down logs while investigating issues, e.g. by allowing to search all the log entries for a specific Machine object, eventually across different controllers/reconciler logs.
• Cluster API developers MUST use logs! As Cluster API contributors you are not only the ones that implement logs, but also the first users of them. Use it! Provide feedback!

Upstream Alignment

Kubernetes defines a set of logging conventions, as well as tools and libraries for logging.

Cluster API should align to those guidelines and use those tools as much as possible.

Continuous improvement

The foundational items of Cluster API logging are:

• Support for structured logging in all the Cluster API controllers (see log format).
• Using contextual logging (see contextual logging).
• Adding a minimal set of key/value pairs in the logger at the beginning of each reconcile loop, so all the subsequent log entries will inherit them (see key value pairs).

Starting from the above foundations, then the long tail of small improvements will consist of following activities:

• Improve consistency of additional key/value pairs added by single log entries (see key value pairs).
• Improve log messages (see log messages).
• Improve consistency of log levels (see log levels).

Log Format

Controllers MUST provide support for structured logging and for the JSON output format; quoting the Kubernetes documentation, these are the key elements of this approach:

• Separate a log message from its arguments.
• Treat log arguments as key-value pairs.
• Be easily parsable and queryable.

Cluster API uses all the tooling provided by the Kubernetes community to implement structured logging: Klog, a logr wrapper that works with controller runtime, and other utils for exposing flags in the controller’s main.go.

Ideally, in a future release of Cluster API we will make JSON output format the default format for all the Cluster API controllers (currently the default is still text format).

Contextual logging

Contextual logging is the practice of using a log stored in the context across the entire chain of calls of a reconcile action. One of the main advantages of this approach is that key value pairs which are added to the logger at the beginning of the chain are then inherited by all the subsequent log entries created down the chain.

Contextual logging is also embedded in controller runtime; In Cluster API we use contextual logging via controller runtime’s LoggerFrom(ctx) and LoggerInto(ctx, log) primitives and this ensures that:

• The logger passed to each reconcile call has a unique reconcileID, so all the logs being written during a single reconcile call can be easily identified (note: controller runtime also adds other useful key value pairs by default).
• The logger has a key value pair identifying the objects being reconciled,e.g. a Machine Deployment, so all the logs impacting this object can be easily identified.

Cluster API developer MUST ensure that:

• The logger has a set of key value pairs identifying the hierarchy of objects the object being reconciled belongs to, e.g. the Cluster a Machine Deployment belongs to, so it will be possible to drill down logs for related Cluster API objects while investigating issues.

Key/Value Pairs

One of the key elements of structured logging is key-value pairs.

Having consistent key value pairs is a requirement for ensuring readability and for providing support for searching and correlating lines across logs.

A set of good practices for defining key value pairs is defined in the Kubernetes Guidelines, and one of the above practices is really important for Cluster API developers

• Developers MUST use klog.KObj or klog.KRef functions when logging key value pairs for Kubernetes objects, thus ensuring a key value pair representing a Kubernetes object is formatted consistently in all the logs.
• Developers MUST use consistent log keys:
  • kinds should be written in upper camel case, e.g. MachineDeployment, MachineSet
    • Note: we cannot use lower camel case for kinds consistently because there is no way to automatically calculate the correct log key for provider CRDs like AWSCluster
  • all other keys should use lower camel case, e.g. resourceVersion, oldReplicas to align to Kubernetes log conventions

Please note that, in order to ensure logs can be easily searched it is important to ensure consistency for the following key value pairs (in order of importance):

• Key value pairs identifying the object being reconciled, e.g. a MachineDeployment.
• Key value pairs identifying the hierarchy of objects being reconciled, e.g. the Cluster a MachineDeployment belongs to.
• Key value pairs identifying side effects on other objects, e.g. while reconciling a MachineDeployment, the controller creates a MachineSet.
• Other Key value pairs.

Notably, over time in CAPI we are also standardizing usage of other key value pairs to improve consistency when reading logs, e.g.

• key reason MUST be used when adding details about WHY a change happened.
• key diff MUST be used when documenting the diff in an object that either lead to a change, or that is resulting from a change.

Log Messages

• A Message MUST always start with a capital letter.
• Period at the end of a message MUST be omitted.
• Always prefer logging before the action, so in case of errors there will be an immediate, visual correlation between the action log and the corresponding error log; While logging before the action, log verbs should use the -ing form.
• Ideally log messages should surface a different level of detail according to the target log level (see log levels for more details).
• If Kubernetes resource name is used in log messages, it MUST be used as is, For example Reconciling DockerMachineTemplate
• If an API field name is used in log messages, the entire path MUST be used and field names MUST capitalized like in the API (not as in the golang type). For example Waiting for spec.providerID to be set
• If a log message is about a controlled or a referenced object, e.g. Machine controller performing an action on MachineSet, the message MUST contain the Kind of the controlled/referenced object and its namespace/name, for example Created MachineSet default/foo-bar
  • The controlled/referenced object MUST also be added as a key value pair (see guidelines above)

Log Levels

Kubernetes provides a set of recommendations for log levels; as a small integration on the above guidelines we would like to add:

• Logs at the lower levels of verbosity (<=3) are meant to document “what happened” by describing how an object status is being changed by controller/reconcilers across subsequent reconciliations; as a rule of thumb, it is reasonable to assume that a person reading those logs has a deep knowledge of how the system works, but it should not be required for those persons to have knowledge of the codebase.
• Logs at higher levels of verbosity (>=4) are meant to document “how it happened”, providing insight on thorny parts of the code; a person reading those logs usually has deep knowledge of the codebase.
• Don’t use verbosity higher than 5.

We are using log level 2 as a default verbosity for all core Cluster API controllers as recommended by the Kubernetes guidelines.

Trade-offs

When developing logs there are operational trade-offs to take into account, e.g. verbosity vs space allocation, user readability vs machine readability, maintainability of the logs across the code base.

A reasonable approach for logging is to keep things simple and implement more log verbosity selectively and only on thorny parts of code. Over time, based on feedback from SRE/developers, more logs can be added to shed light where necessary.

Developing and testing logs

Our Tilt setup offers a batteries-included log suite based on alloy, Loki and Grafana.

We are working to continuously improving this experience, allowing Cluster API developers to use logs and improve them as part of their development process.

For the best experience exploring the logs using Tilt:

1. Set --logging-format=json.
2. Set a high log verbosity, e.g. v=5.
3. Enable alloy, Loki, and Grafana under deploy_observability.

A minimal example of a tilt-settings.yaml file that deploys a ready-to-use logging suite looks like:

deploy_observability:
  - alloy
  - loki
  - grafana
enable_providers:
  - docker
  - kubeadm-bootstrap
  - kubeadm-control-plane
extra_args:
  core:
    - "--logging-format=json"
    - "--v=5"
  docker:
    - "--v=5"
    - "--logging-format=json"
  kubeadm-bootstrap:
    - "--v=5"
    - "--logging-format=json"
  kubeadm-control-plane:
    - "--v=5"
    - "--logging-format=json"

The above options can be combined with other settings from our Tilt setup. Once Tilt is up and running with these settings users will be able to browse logs using the Grafana Explore UI.

This will normally be available on localhost:3000. To explore logs from Loki, open the Explore interface for the DataSource ‘Loki’. This link should work as a shortcut with the default Tilt settings.

Example queries

In the Log browser the following queries can be used to browse logs by controller, and by specific Cluster API objects. For example:

{app="capi-controller-manager"} | json 

Will return logs from the capi-controller-manager which are parsed in json. Passing the query through the json parser allows filtering by key-value pairs that are part of nested json objects. For example .cluster.name becomes cluster_name.

{app="capi-controller-manager"} | json | Cluster_name="my-cluster"

Will return logs from the capi-controller-manager that are associated with the Cluster my-cluster.

{app="capi-controller-manager"} | json | Cluster_name="my-cluster" | v <= 2

Will return logs from the capi-controller-manager that are associated with the Cluster my-cluster with log level <= 2.

{app="capi-controller-manager"} | json | Cluster_name="my-cluster" reconcileID="6f6ad971-bdb6-4fa3-b803-xxxxxxxxxxxx"

Will return logs from the capi-controller-manager, associated with the Cluster my-cluster and the Reconcile ID 6f6ad971-bdb6-4fa3-b803-xxxxxxxxxxxx. Each reconcile loop will have a unique Reconcile ID.

{app="capi-controller-manager"} | json | Cluster_name="my-cluster" reconcileID="6f6ad971-bdb6-4fa3-b803-ef81c5c8f9d0" controller="cluster" | line_format "{{ .msg }}"

Will return logs from the capi-controller-manager, associated with the Cluster my-cluster and the Reconcile ID 6f6ad971-bdb6-4fa3-b803-xxxxxxxxxxxx it further selects only those logs which come from the Cluster controller. It will then format the logs so only the message is displayed.

{app=~"capd-controller-manager|capi-kubeadm-bootstrap-controller-manager|capi-kubeadm-control-plane-controller-manager"} | json | Cluster_name="my-cluster" Machine_name="my-cluster-linux-worker-1" | line_format "{{.controller}} {{.msg}}"

Will return the logs from four CAPI providers - the Core provider, Kubeadm Control Plane provider, Kubeadm Bootstrap provider and the Docker infrastructure provider. It filters by the cluster name and the machine name and then formats the log lines to show just the source controller and the message. This allows us to correlate logs and see actions taken by each of these four providers related to the machine my-cluster-linux-worker-1.

For more information on formatting and filtering logs using Grafana and Loki see:

• log queries

What about providers

Cluster API providers are developed by independent teams, and each team is free to define their own processes and conventions.

However, given that SRE/developers looking at logs are often required to look both at logs from core CAPI and providers, we encourage providers to adopt and contribute to the guidelines defined in this document.

It is also worth noting that the foundational elements of the approach described in this document are easy to achieve by leveraging default Kubernetes tooling for logging.

Testing Cluster API

This document presents testing guidelines and conventions for Cluster API.

IMPORTANT: improving and maintaining this document is a collaborative effort, so we are encouraging constructive feedback and suggestions.

Unit tests

Unit tests focus on individual pieces of logic - a single func - and don’t require any additional services to execute. They should be fast and great for getting the first signal on the current implementation, but unit tests have the risk of allowing integration bugs to slip through.

In Cluster API most of the unit tests are developed using go test, gomega and the fakeclient; however using fakeclient is not suitable for all the use cases due to some limitations in how it is implemented. In some cases contributors will be required to use envtest. See the quick reference below for more details.

Mocking external APIs

In some cases when writing tests it is required to mock external API, e.g. etcd client API or the AWS SDK API.

This problem is usually well scoped in core Cluster API, and in most cases it is already solved by using fake implementations of the target API to be injected during tests.

Instead, mocking is much more relevant for infrastructure providers; in order to address the issue some providers can use simulators reproducing the behaviour of a real infrastructure providers (e.g CAPV); if this is not possible, a viable solution is to use mocks (e.g CAPA).

Generic providers

When writing tests core Cluster API contributors should ensure that the code works with any providers, and thus it is required to not use any specific provider implementation. Instead, the so-called generic providers e.g. “GenericInfrastructureCluster” should be used because they implement the plain Cluster API contract. This prevents tests from relying on assumptions that may not hold true in all cases.

Please note that in the long term we would like to improve the implementation of generic providers, centralizing the existing set of utilities scattered across the codebase, but while details of this work will be defined do not hesitate to reach out to reviewers and maintainers for guidance.

Integration tests

Integration tests are focused on testing the behavior of an entire controller or the interactions between two or more Cluster API controllers.

In Cluster API, integration tests are based on envtest and one or more controllers configured to run against the test cluster.

With this approach it is possible to interact with Cluster API almost like in a real environment, by creating/updating Kubernetes objects and waiting for the controllers to take action. See the quick reference below for more details.

Also in case of integration tests, considerations about mocking external APIs and usage of generic providers apply.

Fuzzing tests

Fuzzing tests automatically inject randomly generated inputs, often invalid or with unexpected values, into functions to discover vulnerabilities.

Two different types of fuzzing are currently being used on the Cluster API repository:

Fuzz testing for API conversion

Cluster API uses Kubernetes’ conversion-gen to automate the generation of functions to convert our API objects between versions. These conversion functions are tested using the FuzzTestFunc util in our conversion utils package. For more information about these conversions see the API conversion code walkthrough in our video walkthrough series.

OSS-Fuzz continuous fuzzing

Parts of the CAPI code base are continuously fuzzed through the OSS-Fuzz project. Issues found in these fuzzing tests are reported to Cluster API maintainers and surfaced in issues on the repo for resolution. To read more about the integration of Cluster API with OSS Fuzz see the 2022 Cluster API Fuzzing Report.

Test maintainability

Tests are an integral part of the project codebase.

Cluster API maintainers and all the contributors should be committed to help in ensuring that tests are easily maintainable, easily readable, well documented and consistent across the code base.

In light of continuing improving our practice around this ambitious goal, we are starting to introduce a shared set of:

• Builders (sigs.k8s.io/cluster-api/util/test/builder), allowing to create test objects in a simple and consistent way.
• Matchers (sigs.k8s.io/controller-runtime/pkg/envtest/komega), improving how we write test assertions.