Add hosted executor as agentless server implementation

[brandond]

Inspired by a discussion with @glennpratt

Proposed Changes

Add a new hosted executor implementation that runs control-plane pods as Deployment/StatefulSet in a hosting cluster, with etcd data stored on PVs. This executor is used when RKE2 is started with --disable-agent. The "agent" is containerd and the kubelet, so without those around we need something else to run the pods - and it might as well be Kubernetes again.

This is very similar to k3k, except in this case the nodes are expected to be normal RKE2 nodes; it is only the control-plane that is virtualized as pods.

Todo:

• Figure out how to handle scaling up/down the etcd StatefulSet - need to manage cluster membership
• Replace the iptables forwarder container in the apiserver pods with a thin supervisor process to handle the remotedialer tunnel mesh
• Dynamic config reloading so that the process does not need to be restarted to sync config
• Watch apiserver service and sync IP into apiserver advertise-address setting when changed
• IPv6 support; validate that apiserver service IP AF == primary service CIDR AF
• rke2 supervisor instead of rke2 server --disable-agent?
• Additional configuration for component topology spread constraints
• Additional configuration for etcd PVCs

Work for a CAPI control-plane operator to handle:

• RBAC/Deployment for supervisor (as per example yaml below)
• Day 2 Ops - snapshot save/restore, cert rotation, scale up/down, etc.
  rke2 certificate/etcd-snapshot/secrets-encrypt should work as-is within supervisor pod, or remotely with correct token and server address.

Here's an example deployment manifest:

Notes:

• This needs the net.ipv4.ip_forward sysctl added to the kubelet's allowed sysctl list; both k3s and rke2 automatically add this when servicelb is enabled (which it is not by default on rke2).
• This uses a LoadBalancer service to set the address advertised by the apiserver. Only LoadBalancers that set an IP in the service.status.ingress are supported; LoadBalancers that set a hostname will not work as the apiserver will only advertise IPs. Currently only the first IP is advertised.
• In this example, the RKE2 supervisor and apiserver are run on non-standard ports so that K3s's ServiceLB can be used to expose the cluster, without conflicting with the K3s apiserver port. If you are using Kube-VIP or MetalLB or something else that exposes a real VIP, you can use the standard ports.

apiVersion: v1
kind: Namespace
metadata:
  name: test-namespace
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-01-supervisor-cluster-admin
subjects:
- kind: ServiceAccount
  name: cluster-01-supervisor
  namespace: test-namespace
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cluster-01-supervisor
  namespace: test-namespace
---
apiVersion: v1
kind: Secret
metadata:
  name: cluster-01-supervisor-config
  namespace: test-namespace
type: Opaque
stringData:
  token.yaml: |
    token: token
---
apiVersion: v1
kind: Service
metadata:
  name: cluster-01-supervisor
  namespace: test-namespace
spec:
  type: ClusterIP
  selector:
    app: rke2
    tier: control-plane
    component: supervisor
    rke2.cattle.io/cluster-name: cluster-01
  ports:
  - name: supervisor
    port: 9355
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-01-supervisor
  namespace: test-namespace
  labels:
    app: rke2
    rke2.cattle.io/cluster-name: cluster-01
spec:
  replicas: 2
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      app: rke2
      tier: control-plane
      component: supervisor
      rke2.cattle.io/cluster-name: cluster-01
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: rke2
        tier: control-plane
        component: supervisor
        rke2.cattle.io/cluster-name: cluster-01
    spec:
      serviceAccountName: cluster-01-supervisor
      terminationGracePeriodSeconds: 30
      volumes:
        - name: config-secret
          secret:
            secretName: cluster-01-supervisor-config
      containers:
        - name: supervisor
          image: docker.io/brandond/rke2-supervisor:latest
          args:
            - server
            - --egress-selector-mode=pod
            - --disable-agent
            - --debug
          env:
            - name: CATTLE_ELECTION_LEASE_DURATION
              value: "15s"
            - name: CATTLE_ELECTION_RENEW_DEADLINE
              value: "10s"
            - name: RKE2_DEPLOYMENT_REPLICAS
              value: "2"
            - name: RKE2_STATEFULSET_REPLICAS
              value: "3"
            - name: RKE2_CLUSTER_SERVICETYPE
              value: "LoadBalancer"
            - name: RKE2_APISERVER_PORT
              value: "6453"
            - name: RKE2_SUPERVISOR_PORT
              value: "9355"
            - name: RKE2_CLUSTER_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['rke2.cattle.io/cluster-name']
          imagePullPolicy: Always
          volumeMounts:
            - name: config-secret
              readOnly: true
              mountPath: /etc/rancher/rke2/config.yaml.d
          readinessProbe:
            exec:
              command:
                - kubectl
                - get
                - --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig
                - --server=https://localhost:$(RKE2_SUPERVISOR_PORT)
                - --raw=/v1-rke2/readyz

From the outer cluster (k3s):

brandond@dev01:~$ kubectl get pod,service -n test-namespace
NAME                                                       READY   STATUS    RESTARTS   AGE
pod/cluster-01-cloud-controller-manager-6dbdcdf946-7hmpf   1/1     Running   0          5m6s
pod/cluster-01-cloud-controller-manager-6dbdcdf946-d55zx   1/1     Running   0          5m6s
pod/cluster-01-etcd-0                                      1/1     Running   0          5m50s
pod/cluster-01-etcd-1                                      1/1     Running   0          5m43s
pod/cluster-01-etcd-2                                      1/1     Running   0          5m37s
pod/cluster-01-kube-apiserver-55bc8b969c-48jhz             2/2     Running   0          5m34s
pod/cluster-01-kube-apiserver-55bc8b969c-l7552             2/2     Running   0          5m34s
pod/cluster-01-kube-controller-manager-8bdb46c4c-qh7bj     1/1     Running   0          5m8s
pod/cluster-01-kube-controller-manager-8bdb46c4c-sv4jw     1/1     Running   0          5m8s
pod/cluster-01-kube-scheduler-554dcd8c9c-jzdtr             1/1     Running   0          5m8s
pod/cluster-01-kube-scheduler-554dcd8c9c-mvk8n             1/1     Running   0          5m8s
pod/cluster-01-supervisor-5684995c97-f7ghg                 0/1     Running   0          5m58s
pod/cluster-01-supervisor-5684995c97-ww2sb                 1/1     Running   0          5m58s

NAME                                TYPE           CLUSTER-IP      EXTERNAL-IP             PORT(S)                         AGE
service/cluster-01-etcd             ClusterIP      None            <none>                  2379/TCP,2380/TCP,2381/TCP      5m50s
service/cluster-01-kube-apiserver   LoadBalancer   10.43.179.151   172.17.0.4,172.17.0.5   6453:32213/TCP,9355:30316/TCP   5m52s
service/cluster-01-supervisor       ClusterIP      10.43.14.29     <none>                  9355/TCP                        5m58s

RKE2 agents were joined against the apiserver loadbalancer address:

token: token
server: https://172.17.0.4:9355

From the inner cluster (rke2) - note that there are no control-plane/etcd nodes; the control-plane is completely isolated from agents:

/ # export KUBECONFIG=/etc/rancher/rke2/rke2.yaml

/ # kubectl get node -o wide
NAME                         STATUS   ROLES    AGE     VERSION          INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
rke2-agent-001.example.com   Ready    <none>   3m1s    v1.33.1+rke2r1   172.17.0.6    <none>        Ubuntu 22.04.5 LTS   6.8.0-1027-aws   containerd://2.0.5-k3s1
rke2-agent-002.example.com   Ready    <none>   2m47s   v1.33.1+rke2r1   172.17.0.7    <none>        Ubuntu 22.04.5 LTS   6.8.0-1027-aws   containerd://2.0.5-k3s1

/ # kubectl get service -A
NAMESPACE     NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       kubernetes                                ClusterIP   10.43.0.1       <none>        443/TCP         2m50s
kube-system   rke2-coredns-rke2-coredns                 ClusterIP   10.43.0.10      <none>        53/UDP,53/TCP   2m10s
kube-system   rke2-ingress-nginx-controller-admission   ClusterIP   10.43.117.148   <none>        443/TCP         70s
kube-system   rke2-metrics-server                       ClusterIP   10.43.240.170   <none>        443/TCP         77s

/ # kubectl get pod -A
NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-canal-88hvw                           0/1     Completed   0          2m27s
kube-system   helm-install-rke2-coredns-pdkz8                         0/1     Completed   0          2m27s
kube-system   helm-install-rke2-ingress-nginx-xpc7j                   0/1     Completed   0          2m27s
kube-system   helm-install-rke2-metrics-server-br7f2                  0/1     Completed   0          2m27s
kube-system   helm-install-rke2-runtimeclasses-wcpln                  0/1     Completed   0          2m26s
kube-system   helm-install-rke2-snapshot-controller-crd-g4pkc         0/1     Completed   0          2m26s
kube-system   helm-install-rke2-snapshot-controller-rzcbk             0/1     Completed   1          2m26s
kube-system   kube-proxy-rke2-agent-001.example.com                   1/1     Running     0          2m33s
kube-system   kube-proxy-rke2-agent-002.example.com                   1/1     Running     0          2m18s
kube-system   rke2-canal-ghslx                                        2/2     Running     0          2m18s
kube-system   rke2-canal-xp69m                                        2/2     Running     0          2m17s
kube-system   rke2-coredns-rke2-coredns-autoscaler-59688fcc54-4cbkx   1/1     Running     0          2m22s
kube-system   rke2-coredns-rke2-coredns-f79c8888f-q4x64               1/1     Running     0          89s
kube-system   rke2-coredns-rke2-coredns-f79c8888f-z8hzb               1/1     Running     0          2m22s
kube-system   rke2-ingress-nginx-controller-26f28                     1/1     Running     0          82s
kube-system   rke2-ingress-nginx-controller-725w6                     1/1     Running     0          82s
kube-system   rke2-metrics-server-544fd45cb9-nv8l9                    1/1     Running     0          89s
kube-system   rke2-snapshot-controller-696989ffdd-8nf64               1/1     Running     0          88s

/ # kubectl get lease -A
NAMESPACE         NAME                                   HOLDER                                                                                      AGE
kube-node-lease   rke2-agent-001.example.com             rke2-agent-001.example.com                                                                  2m38s
kube-node-lease   rke2-agent-002.example.com             rke2-agent-002.example.com                                                                  2m32s
kube-system       apiserver-26c557qhbly4skhpesnzg4ua7y   apiserver-26c557qhbly4skhpesnzg4ua7y_866e16b6-f405-42f6-b716-4d206e56bf79                   3m19s
kube-system       apiserver-zm2xumeuwuflrjzpnyzvhj4sxm   apiserver-zm2xumeuwuflrjzpnyzvhj4sxm_742b22db-6199-4487-9e5a-04dfb0695e3c                   3m20s
kube-system       kube-controller-manager                cluster-01-kube-controller-manager-8bdb46c4c-sv4jw_fa4226fb-4178-4ca3-a38c-9c03740205bf     3m3s
kube-system       kube-scheduler                         cluster-01-kube-scheduler-554dcd8c9c-jzdtr_633d3fb1-a5af-45d1-ac94-ffb76bfa17bd             3m2s
kube-system       rke2                                   cluster-01-supervisor-5684995c97-ww2sb                                                      3m
kube-system       rke2-cloud-controller-manager          cluster-01-cloud-controller-manager-6dbdcdf946-7hmpf_13a06e39-90f8-4e9b-9f5c-302f3cf573f9   2m59s
kube-system       rke2-etcd                              cluster-01-supervisor-5684995c97-ww2sb                                                      3m
kube-system       rke2-ingress-nginx-leader              rke2-ingress-nginx-controller-725w6                                                         81s
kube-system       snapshot-controller-leader             rke2-snapshot-controller-696989ffdd-8nf64                                                   99s

/ # kubectl get endpointslice -n default kubernetes
NAME         ADDRESSTYPE   PORTS   ENDPOINTS    AGE
kubernetes   IPv4          6453    172.17.0.4   4m21s

Types of Changes

enhancement

Verification

Testing

Linked Issues

• Respect k3s experimental agentless server option #8289

User-Facing Change

Further Comments

[codecov-commenter]

Codecov Report

❌ Patch coverage is 1.44033% with 958 lines in your changes missing coverage. Please review.
✅ Project coverage is 15.48%. Comparing base (4d9368a) to head (e48f48f).
⚠️ Report is 409 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/executor/hosted/generators.go	0.00%	445 Missing ⚠️
pkg/executor/hosted/hosted.go	0.00%	317 Missing ⚠️
pkg/rke2/rke2_linux.go	4.34%	65 Missing and 1 partial ⚠️
pkg/executor/hosted/controllers.go	0.00%	61 Missing ⚠️
pkg/executor/hosted/driver.go	0.00%	42 Missing ⚠️
pkg/cli/defaults/defaults.go	46.66%	4 Missing and 4 partials ⚠️
pkg/hardening/profile/profile.go	33.33%	8 Missing ⚠️
pkg/cli/cmds/profile_linux.go	0.00%	4 Missing ⚠️
pkg/executor/staticpod/staticpod.go	0.00%	3 Missing ⚠️
pkg/rke2/rke2_windows.go	0.00%	3 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8301      +/-   ##
==========================================
- Coverage   18.37%   15.48%   -2.90%     
==========================================
  Files          47       52       +5     
  Lines        4859     5801     +942     
==========================================
+ Hits          893      898       +5     
- Misses       3923     4855     +932     
- Partials       43       48       +5     

Flag	Coverage Δ
inttests	6.78% <0.00%> (-1.81%)	⬇️
unittests	11.45% <1.44%> (-2.13%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.