Skip to content

Commit

Permalink
fixup! K8s cluster robustness features (#414)
Browse files Browse the repository at this point in the history
Signed-off-by: Hannes Baum <[email protected]>
  • Loading branch information
cah-hbaum committed Dec 7, 2023
1 parent 7a879e7 commit abe580f
Showing 1 changed file with 16 additions and 25 deletions.
41 changes: 16 additions & 25 deletions Standards/scs-0215-v1-robustness-features.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ To mitigate this scenario, the etcd keyspace can be compacted. Additionally, an
gives back disk space to the underlying file system and can help bring the cluster back into an operable state, if it
ran out of space earlier.

This can be achieved by providing the necessary flags/parameters to etcd, either via the KubeadmControlPlane or in the
etcd keyspace maintenance can be achieved by providing the necessary flags/parameters to etcd, either via the KubeadmControlPlane or in the
configuration file of the etcd cluster, if it is managed independent of the Kubernetes cluster.
Possible flags, that can be set for this feature, are:

Expand All @@ -118,7 +118,7 @@ meaning that the previous year should only have 1 backup, but the current week s
Information about the backup process can be found in the etcd documentation:
[Upgrade etcd](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/)

### CA expiration avoidance
### Certificate rotation

In order to secure the communication of a Kubernetes cluster, (TLS) certificates signed by a controlled
Certificate Authority (CA) can be used.
Expand All @@ -130,8 +130,9 @@ kubelet server certificate. Another recommendation is to set `serverTLSBootstrap
and rotation of the certificate for the kubelet according to the documentation.

A clusters certificates can either be rotated by updating the cluster, which according to the Kubernetes documentation
automatically renews the certificates, or by running the kubeadm certs renew command for the certificates that need to
updated, which can be seen in the following inline code
automatically renews the certificates.
Another method would be the manual renewal, which can be done through multiple methods, depending on the K8s cluster
used. An example for a default K8s cluster would be to execute the command

```bash
kubeadm certs renew all
Expand All @@ -140,7 +141,7 @@ kubeadm certs renew all
Since clusters conformant with the SCS standards would probably be updated within the time period described in the
standard [SCS-0210-v2](https://github.com/SovereignCloudStack/standards/tree/main/Standards/scs-0210-v2-k8s-version-policy.md),
this rotation can probably be assumed to happen. Nevertheless, the alternative can still be mentioned in the standard.
Additionally, the Certificate Signing Request (CSR) needs to be approved manually due to security reasons with the commands
Additionally, the Certificate Signing Request (CSR) need to be approved manually due to security reasons with the commands

```bash
kubectl get csr
Expand All @@ -152,8 +153,8 @@ this would be the [Kubelet CSR approver](https://github.com/postfinance/kubelet-
a K8s cluster and requires `serverTLSBootstrap` to be set to true. Other controllers with a similar functionality might
have other specific requirements, which won't be explored in this document.

Another problem is that the CA might expire. Unfortunately, kubeadm doesn't have any tooling
at the moment to renew the CA. Instead, there is documentation for manually rotating the CA, which can be found
Another problem is that the CA might expire. Unfortunately, not all K8s tools have functionality to renew these
certificates with the help of commands. Instead, there is documentation for manually rotating the CA, which can be found
under [Manual rotation of ca certificate](https://kubernetes.io/docs/tasks/tls/manual-rotation-of-ca-certificates/).

Further information can be found in the Kubernetes documentation:
Expand All @@ -168,16 +169,16 @@ areas.

### Kube-API rate limiting

The number of requests send to the kube-api or Kubernetes API server SHOULD be limited
The number of requests send to the kube-api or Kubernetes API server MUST be limited
in order to protect the server against outages, deceleration or malfunctions due to an
overload of requests.
In order to do so, at least the following parameters SHOULD be set on a Kubernetes cluster:
In order to do so, at least the following parameters MUST be set on a Kubernetes cluster:

* max-requests-inflight
* max-mutating-requests-inflight
* min-request-timeout

Values for these flags/parameters MUST be adapted to the needs of the environment and
Values for these flags/parameters SHOULD be adapted to the needs of the environment and
the expected load.

A cluster MUST also activate and configure a Ratelimit admission controller.
Expand Down Expand Up @@ -219,28 +220,18 @@ These numbers can be adapted to the security setup and concerns like storage or
usage. It is also RECOMMENDED to encrypt the backups in order to secure them further.
How this is done is up to the operator.
### CA expiration avoidance
### Certificate rotation
It should be avoided, that certificates expire either on the whole cluster or for single components.
To avoid this scenario, certificates SHOULD be rotated regularly; in the
case of SCS, we REQUIRE at least a yearly certificate rotation.
To achieve a complete certificate rotation, the parameters `serverTLSBootstrap` and `rotateCertificates` MUST be set.
To achieve a complete certificate rotation, the parameters `serverTLSBootstrap` and `rotateCertificates`
MUST be set in the kubelet configuration.

The certificates can be rotated by either updating the Kubernetes cluster, which automatically
renews certificates, or by manually renewing them with the command
renews certificates, or by manually renewing them. How this is done is dependent on the used K8s cluster.

```bash
kubeadm certs renew all
```

After this, new CSRs MUST be approved manually, normally done with

```bash
kubectl get csr
kubectl certificate approve <CSR>
```

or be approved with a third-party controller, e.g. the [kubelet-csr-approver](https://github.com/postfinance/kubelet-csr-approver).
After this, new CSRs MUST be approved manually or with a third-party controller, e.g. the [kubelet-csr-approver](https://github.com/postfinance/kubelet-csr-approver).

It is also RECOMMENDED to renew the CA regularly to avoid an expiration of the CA.
This standard doesn't set a timeline for this, since it is dependent on the CA.
Expand Down

0 comments on commit abe580f

Please sign in to comment.