Kubernetes Multitenant clusters via CAPI

[obriensystems]

For the CAPI management server - install via KIND on Docker Desktop on a mac (M4, M4pro)

• see Kubernetes on Macbook pro 16 M4 Max on multiple VMWare Fusion 13 Ubuntu 24.04 VMs #75
• see Kubernetes on Mac Mini M4 Pro Machines using Thunderbolt 5 networking on VMWare Fusion 13 Ubuntu 24.04 VMs #110
• see Kubernetes Multitenant clusters via CAPI pipeline#13
• https://cluster-api.sigs.k8s.io/user/quick-start
• https://kind.sigs.k8s.io/docs/user/quick-start

[ $(uname -m) = arm64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.27.0/kind-darwin-arm64
chmod +x ./kind
mv ./kind ~/opt/kind

[obriensystems]

create kind cluster as per https://cluster-api.sigs.k8s.io/user/quick-start

rien@mbp8 src % kind create cluster   
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.32.2) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
en@mbp8 src % kubectl cluster-info --context kind-kind
Kubernetes control plane is running at https://127.0.0.1:52942
CoreDNS is running at https://127.0.0.1:52942/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
ien@mbp8 src % kubectl get nodes --context kind-kind
NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   47s   v1.32.2

[obriensystems]

Install clusterctl

..n@mbp8 ObrienlabsDev % curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.9.4/clusterctl-darwin-amd64 -o clusterctl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 77.3M  100 77.3M    0     0  36.4M      0  0:00:02  0:00:02 --:--:-- 47.2M
..ien@mbp8 ObrienlabsDev % chmod +x ./clusterctl
..ien@mbp8 ObrienlabsDev % sudo mv ./clusterctl /usr/local/bin/clusterctl
.en@mbp8 ObrienlabsDev % clusterctl version
clusterctl version: &version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.4", GitCommit:"79e6731b0d20a0ca2dad5e4f96d2a532aa82b54b", GitTreeState:"clean", BuildDate:"2025-01-21T17:04:26Z", GoVersion:"go1.22.10", Compiler:"gc", Platform:"darwin/amd64"}

New clusterctl version available: v1.9.4 -> v1.9.5
sigs.k8s.io/cluster-api

[obriensystems]

initialize clusterctl

.,..en@mbp8 ObrienlabsDev % kubectl config get-contexts
                               
*         kind-kind                                         kind-kind                                         kind-kind                                         

,,,ien@mbp8 ObrienlabsDev % clusterctl init
Fetching providers
Installing cert-manager version="v1.16.2"
Waiting for cert-manager to be available...
Installing provider="cluster-api" version="v1.9.5" targetNamespace="capi-system"
Installing provider="bootstrap-kubeadm" version="v1.9.5" targetNamespace="capi-kubeadm-bootstrap-system"
Installing provider="control-plane-kubeadm" version="v1.9.5" targetNamespace="capi-kubeadm-control-plane-system"

Your management cluster has been initialized successfully!

You can now create your first workload cluster by running the following:

  clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f -

New clusterctl version available: v1.9.4 -> v1.9.5
sigs.k8s.io/cluster-api

export CLUSTER_TOPOLOGY=true

[obriensystems]

setup providers
https://cluster-api.sigs.k8s.io/user/quick-start#initialization-for-common-providers

start with docker

rien@mbp8 ObrienlabsDev % clusterctl init --infrastructure docker
Fetching providers
Skipping installing cert-manager as it is already installed
Installing provider="infrastructure-docker" version="v1.9.5" targetNamespace="capd-system"

New clusterctl version available: v1.9.4 -> v1.9.5
sigs.k8s.io/cluster-api

[obriensystems]

create cluster

ien@mbp8 ObrienlabsDev % clusterctl generate cluster obl1 --kubernetes-version v1.31.3 | kubectl apply -f -
Error: failed to read "cluster-template.yaml" from provider's repository "infrastructure-docker": failed to download files from GitHub release v1.9.5: failed to get file "cluster-template.yaml" from "v1.9.5" release
error: no objects passed to apply

[obriensystems]

upgrade clusterctl to attempt successful cluster creation

n@mbp8 ObrienlabsDev % clusterctl upgrade plan
Checking if cert-manager needs upgrade...
Cert-Manager is already up to date

Checking new release availability...

Latest release available for the v1beta1 API Version of Cluster API (contract):

NAME                    NAMESPACE                           TYPE                     CURRENT VERSION   NEXT VERSION
bootstrap-kubeadm       capi-kubeadm-bootstrap-system       BootstrapProvider        v1.9.5            Already up to date
control-plane-kubeadm   capi-kubeadm-control-plane-system   ControlPlaneProvider     v1.9.5            Already up to date
cluster-api             capi-system                         CoreProvider             v1.9.5            Already up to date
infrastructure-docker   capd-system                         InfrastructureProvider   v1.9.5            Already up to date

You are already up to date!
New clusterctl version available: v1.9.4 -> v1.9.5
sigs.k8s.io/cluster-api

en@mbp8 ObrienlabsDev % export CLUSTERCTL_VERSION=v1.9.5
en@mbp8 ObrienlabsDev % curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/${CLUSTERCTL_VERSION}/clusterctl-darwin-arm64 -o clusterctl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 75.2M  100 75.2M    0     0  33.9M      0  0:00:02  0:00:02 --:--:-- 42.5M
en@mbp8 ObrienlabsDev % chmod +x clusterctl
ien@mbp8 ObrienlabsDev % sudo mv clusterctl /usr/local/bin/
en@mbp8 ObrienlabsDev % clusterctl version                                                                                                                
clusterctl version: &version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"068c0f3c8ed3a7e382952f8ae68b648aee240a60", GitTreeState:"clean", BuildDate:"2025-02-18T19:11:43Z", GoVersion:"go1.22.12", Compiler:"gc", Platform:"darwin/arm64"}
en@mbp8 ObrienlabsDev % clusterctl upgrade plan                                                                                                           
Checking if cert-manager needs upgrade...
Cert-Manager will be upgraded from "v1.16.2" to "v1.16.3"
Checking new release availability...
Latest release available for the v1beta1 API Version of Cluster API (contract):

NAME                    NAMESPACE                           TYPE                     CURRENT VERSION   NEXT VERSION
bootstrap-kubeadm       capi-kubeadm-bootstrap-system       BootstrapProvider        v1.9.5            Already up to date
control-plane-kubeadm   capi-kubeadm-control-plane-system   ControlPlaneProvider     v1.9.5            Already up to date
cluster-api             capi-system                         CoreProvider             v1.9.5            Already up to date
infrastructure-docker   capd-system                         InfrastructureProvider   v1.9.5            Already up to date

You are already up to date!

ien@mbp8 ObrienlabsDev % clusterctl generate cluster obl1 --kubernetes-version v1.27.1 | kubectl apply -f -                                                
Error: failed to read "cluster-template.yaml" from provider's repository "infrastructure-docker": failed to download files from GitHub release v1.9.5: failed to get file "cluster-template.yaml" from "v1.9.5" release
error: no objects passed to apply

[obriensystems]

check cluster status

obrien@mbp8 ObrienlabsDev % kubectl get pods -A                                                                                                        
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS         AGE
capd-system                         capd-controller-manager-6f5bfd7bd9-27xwq                         1/1     Running   0                116m
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-66bb86b8b8-65r9x       1/1     Running   0                19h
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-7bd59d5f69-n9wr4   1/1     Running   0                19h
capi-system                         capi-controller-manager-578674dd86-vvg7x                         1/1     Running   0                19h
cert-manager                        cert-manager-767f578ff-9fj2c                                     1/1     Running   21 (3h50m ago)   19h
cert-manager                        cert-manager-cainjector-c7fdb4dbf-6p88t                          1/1     Running   0                19h
cert-manager                        cert-manager-webhook-768bf9d966-6g5hv                            1/1     Running   0                19h
kube-system                         coredns-668d6bf9bc-9q8kp                                         1/1     Running   0                19h
kube-system                         coredns-668d6bf9bc-vq84r                                         1/1     Running   0                19h
kube-system                         etcd-kind-control-plane                                          1/1     Running   0                19h
kube-system                         kindnet-fl2tb                                                    1/1     Running   0                19h
kube-system                         kube-apiserver-kind-control-plane                                1/1     Running   0                19h
kube-system                         kube-controller-manager-kind-control-plane                       1/1     Running   0                19h
kube-system                         kube-proxy-4qmjq                                                 1/1     Running   0                19h
kube-system                         kube-scheduler-kind-control-plane                                1/1     Running   0                19h
local-path-storage                  local-path-provisioner-7dc846544d-jvwjn                          1/1     Running   0                19h

TODO try AWS provider
https://cluster-api.sigs.k8s.io/user/quick-start#initialization-for-common-providers

curl -L https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/download/v2.7.1/clusterawsadm-darwin-arm64 -o clusterawsadm
chmod +x clusterawsadm
sudo mv clusterawsadm /usr/local/bin
clusterawsadm version

export AWS_REGION=us-east-1 # This is used to help encode your environment variables
export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>
export AWS_SESSION_TOKEN=<session-token> # If you are using Multi-Factor Auth.

clusterawsadm bootstrap iam create-cloudformation-stack
export AWS_B64ENCODED_CREDENTIALS=$(clusterawsadm bootstrap credentials encode-as-profile)
clusterctl init --infrastructure aws

[obriensystems]

revisit docker - forgot exports in https://cluster-api.sigs.k8s.io/user/quick-start#required-configuration-for-common-providers

en@mbp8 ObrienlabsDev % export SERVICE_CIDR=["10.96.0.0/12"]
n@mbp8 ObrienlabsDev % export POD_CIDR=["192.168.0.0/16"]
n@mbp8 ObrienlabsDev % export SERVICE_DOMAIN="k8s.test"
en@mbp8 ObrienlabsDev % export POD_SECURITY_STANDARD_ENABLED="false"
en@mbp8 ObrienlabsDev % clusterctl generate cluster capi-quickstart --flavor development \
  --kubernetes-version v1.32.0 \
  --control-plane-machine-count=3 \
  --worker-machine-count=3 \
  > capi-quickstart.yaml

check yaml before applying

[obriensystems]

apply yaml - fix errors

en@mbp8 ObrienlabsDev % kubectl apply -f capi-quickstart.yaml
dockerclustertemplate.infrastructure.cluster.x-k8s.io/quick-start-cluster created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-control-plane created
dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-default-worker-machinetemplate created
dockermachinepooltemplate.infrastructure.cluster.x-k8s.io/quick-start-default-worker-machinepooltemplate created
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/quick-start-default-worker-bootstraptemplate created
Error from server (Forbidden): error when creating "capi-quickstart.yaml": admission webhook "validation.clusterclass.cluster.x-k8s.io" denied the request: spec: Forbidden: can be set only if the ClusterTopology feature flag is enabled
Error from server (Forbidden): error when creating "capi-quickstart.yaml": admission webhook "validation.kubeadmcontrolplanetemplate.controlplane.cluster.x-k8s.io" denied the request: spec: Forbidden: can be set only if the ClusterTopology feature flag is enabled
Error from server (Invalid): error when creating "capi-quickstart.yaml": admission webhook "validation.cluster.cluster.x-k8s.io" denied the request: Cluster.cluster.x-k8s.io "capi-quickstart" is invalid: spec.topology: Forbidden: can be set only if the ClusterTopology feature flag is enabled


it is enabled
michaelobrien@mbp8 ObrienlabsDev % echo $CLUSTER_TOPOLOGY     
true

[obriensystems]

turns out I added the feature flag after I created the cluster

kind create cluster
export CLUSTER_TOPOLOGY=true

need to do the reverse
see
kubernetes-sigs/cluster-api#10764 (comment)

[obriensystems]

delete k8s cluster and kind cluster

ien@mbp8 ObrienlabsDev % kubectl delete -f capi-quickstart.yaml
dockerclustertemplate.infrastructure.cluster.x-k8s.io "quick-start-cluster" deleted
dockermachinetemplate.infrastructure.cluster.x-k8s.io "quick-start-control-plane" deleted
dockermachinetemplate.infrastructure.cluster.x-k8s.io "quick-start-default-worker-machinetemplate" deleted
dockermachinepooltemplate.infrastructure.cluster.x-k8s.io "quick-start-default-worker-machinepooltemplate" deleted
kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io "quick-start-default-worker-bootstraptemplate" deleted
Error from server (NotFound): error when deleting "capi-quickstart.yaml": clusterclasses.cluster.x-k8s.io "quick-start" not found
Error from server (NotFound): error when deleting "capi-quickstart.yaml": kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io "quick-start-control-plane" not found
Error from server (NotFound): error when deleting "capi-quickstart.yaml": clusters.cluster.x-k8s.io "capi-quickstart" not found
en@mbp8 ObrienlabsDev % kind delete cluster
Deleting cluster "kind" ...
Deleted nodes: ["kind-control-plane"]

[obriensystems]

recreate kind cluster after setting CLUSTER_TOPOLOGY

ien@mbp8 ObrienlabsDev % echo $CLUSTER_TOPOLOGY                
true
michaelobrien@mbp8 ObrienlabsDev % kind create cluster                                                                                              
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.32.2) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋
rien@mbp8 ObrienlabsDev % kubectl config get-contexts
CURRENT   NAME                                              CLUSTER                                           AUTHINFO                                          NAMESPACE
          arn:aws:eks:us-east-1:453279094200:cluster/prod   arn:aws:eks:us-east-1:453279094200:cluster/prod   arn:aws:eks:us-east-1:453279094200:cluster/prod   
          docker-desktop                                    docker-desktop                                    docker-desktop                                    
*         kind-kind                                         kind-kind                                         kind-kind                                         
          minikube                                          minikube                                          minikube                                          default
michaelobrien@mbp8 ObrienlabsDev % kubectl cluster-info --context kind-kind
Kubernetes control plane is running at https://127.0.0.1:53435
CoreDNS is running at https://127.0.0.1:53435/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
en@mbp8 ObrienlabsDev % kubectl get nodes
NAME                 STATUS   ROLES           AGE   VERSION
kind-control-plane   Ready    control-plane   41s   v1.32.2