Skip to content

Commit 64a8e79

Browse files
hakmanhakman
committed
Update docs
1 parent 234d39f commit 64a8e79

File tree

1 file changed

+92
-39
lines changed

1 file changed

+92
-39
lines changed

‎docs/operations/karpenter.md‎

Lines changed: 92 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,69 +1,122 @@
11
# Karpenter
22

3-
[Karpenter](https://karpenter.sh) is a Kubernetes-native capacity manager that directly provisions Nodes and underlying instances based on Pod requirements. On AWS, kOps supports managing an InstanceGroup with either Karpenter or an AWS Auto Scaling Group (ASG).
3+
[Karpenter](https://karpenter.sh) is an open-source node lifecycle management project built for Kubernetes.
4+
Adding Karpenter to a Kubernetes cluster can dramatically improve the efficiency and cost of running workloads on that cluster.
5+
6+
On AWS, kOps supports managing an InstanceGroup with either Karpenter or an AWS Auto Scaling Group (ASG).
7+
8+
## Prerequisites
9+
10+
Managed Karpenter requires kOps 1.34+ and that [IAM Roles for Service Accounts (IRSA)](/cluster_spec#service-account-issuer-discovery-and-aws-iam-roles-for-service-accounts-irsa) be enabled for the cluster.
11+
12+
If an older version of Karpenter was installed, it must be uninstalled before installing the new version.
413

514
## Installing
615

7-
If using kOps 1.26 or older, enable the Karpenter feature flag :
16+
### New clusters
817

918
```sh
10-
export KOPS_FEATURE_FLAGS="Karpenter"
11-
```
19+
export KOPS_STATE_STORE="s3://my-state-store"
20+
export KOPS_DISCOVERY_STORE="s3://my-discovery-store"
21+
export NAME="my-cluster.example.com"
22+
export ZONES="eu-central-1a"
1223

13-
Karpenter requires that external permissions for ServiceAccounts be enabled for the cluster. See [AWS IAM roles for ServiceAccounts documentation](/cluster_spec#service-account-issuer-discovery-and-aws-iam-roles-for-service-accounts-irsa) for how to enable this.
24+
kops create cluster --name ${NAME} \
25+
--cloud=aws \
26+
--instance-manager=karpenter \
27+
--discovery-store=${KOPS_DISCOVERY_STORE} \
28+
--zones=${ZONES} \
29+
--yes
30+
31+
kops validate cluster --name ${NAME} --wait=10m
32+
33+
kops export kubeconfig --name ${NAME} --admin
34+
```
1435

1536
### Existing clusters
1637

17-
On existing clusters, you can create a Karpenter InstanceGroup by adding the following to its InstanceGroup spec:
38+
The Karpenter addon must be enabled in the cluster spec:
1839

1940
```yaml
2041
spec:
21-
manager: Karpenter
42+
karpenter:
43+
enabled: true
2244
```
2345
24-
You also need to enable the Karpenter addon in the cluster spec:
46+
To create a Karpenter InstanceGroup, set the following in its InstanceGroup spec:
2547
2648
```yaml
2749
spec:
28-
karpenter:
29-
enabled: true
50+
manager: Karpenter
3051
```
3152
32-
### New clusters
33-
34-
On new clusters, you can simply add the `--instance-manager=karpenter` flag:
53+
### EC2NodeClass and NodePool
3554
3655
```sh
37-
kops create cluster --name mycluster.example.com --cloud aws --networking=amazonvpc --zones=eu-central-1a,eu-central-1b --master-count=3 --yes --discovery-store=s3://discovery-store/
56+
USER_DATA=$(aws s3 cp ${KOPS_STATE_STORE}/${NAME}/igconfig/node/nodes/nodeupscript.sh -)
57+
USER_DATA=${USER_DATA//$'\n'/$'\n '}
58+
59+
kubectl apply -f - <<YAML
60+
apiVersion: karpenter.k8s.aws/v1
61+
kind: EC2NodeClass
62+
metadata:
63+
name: default
64+
spec:
65+
amiFamily: Custom
66+
amiSelectorTerms:
67+
- ssmParameter: /aws/service/canonical/ubuntu/server/24.04/stable/current/amd64/hvm/ebs-gp3/ami-id
68+
- ssmParameter: /aws/service/canonical/ubuntu/server/24.04/stable/current/arm64/hvm/ebs-gp3/ami-id
69+
associatePublicIPAddress: true
70+
tags:
71+
KubernetesCluster: ${NAME}
72+
kops.k8s.io/instancegroup: nodes
73+
k8s.io/role/node: "1"
74+
subnetSelectorTerms:
75+
- tags:
76+
KubernetesCluster: ${NAME}
77+
securityGroupSelectorTerms:
78+
- tags:
79+
KubernetesCluster: ${NAME}
80+
Name: nodes.${NAME}
81+
instanceProfile: nodes.${NAME}
82+
userData: |
83+
${USER_DATA}
84+
YAML
85+
86+
kubectl apply -f - <<YAML
87+
apiVersion: karpenter.sh/v1
88+
kind: NodePool
89+
metadata:
90+
name: default
91+
spec:
92+
template:
93+
spec:
94+
requirements:
95+
- key: kubernetes.io/arch
96+
operator: In
97+
values: ["amd64", "arm64"]
98+
- key: kubernetes.io/os
99+
operator: In
100+
values: ["linux"]
101+
- key: karpenter.sh/capacity-type
102+
operator: In
103+
values: ["on-demand", "spot"]
104+
nodeClassRef:
105+
group: karpenter.k8s.aws
106+
kind: EC2NodeClass
107+
name: default
108+
YAML
38109
```
39110

40111
## Karpenter-managed InstanceGroups
41112

42-
A Karpenter-managed InstanceGroup controls a corresponding Karpenter Provisioner resource. kOps will ensure that the Provisioner is configured with the correct AWS security groups, subnets, and launch templates. Just like with ASG-managed InstanceGroups, you can add labels and taints to Nodes and kOps will ensure those are added accordingly.
43-
44-
Note that not all features of InstanceGroups are supported.
45-
46-
## Subnets
47-
48-
By default, kOps will tag subnets with `kops.k8s.io/instance-group/<intancegroup>: "true"` for each InstanceGroup the subnet is assigned to. If you enable manual tagging of subnets, you have to ensure these tags are added, if not Karpenter will fail to provision any instances.
49-
50-
## Instance Types
51-
52-
If you do not specify a mixed instances policy, only the instance type specified by `spec.machineType` will be used. With Karpenter, one typically wants a wider range of instances to choose from. kOps supports both providing a list of instance types through `spec.mixedInstancesPolicy.instances` and providing instance type requirements through `spec.mixedInstancesPolicy.instanceRequirements`. See (/instance_groups)[InstanceGroup documentation] for more details.
113+
A Karpenter-managed InstanceGroup controls the bootstrap script. kOps will ensure the correct AWS security groups, subnets and permissions.
114+
`EC2NodeClass` and `NodePool` objects must be created by the cluster operator.
53115

54116
## Known limitations
55117

56-
### Karpenter-managed Launch Templates
57-
58-
On EKS, Karpener creates its own launch templates for Provisioners. These launch templates will not work with a kOps cluster for a number of reasons. Most importantly, they do not use supported AMIs and they do not install and configure nodeup, the instance-side kOps component. The Karpenter features that require Karpenter to directly manage launch templates will not be available on kOps.
59-
60-
### Unmanaged Provisioner resources
61-
62-
As mentioned above, kOps will manage a Provisioner resource per InstanceGroup. It is technically possible to create Provsioner resources directly, but you have to ensure that you configure Provisioners according to kOps requirements. As mentioned above, Karpenter-managed launch templates do not work and you have to maintain your own kOps-compatible launch templates.
63-
64-
### Other minor limitations
65-
66-
* Control plane nodes must be provisioned with an ASG, not Karpenter.
67-
* Provisioners will unconditionally use spot with a fallback on ondemand instances.
68-
* Provisioners will unconditionally include burstable instance groups such as the T3 instance family.
69-
* kOps will not allow mixing arm64 and amd64 instances in the same Provider.
118+
* **Upgrade is not supported** from the previous version of managed Karpenter.
119+
* Control plane nodes must be provisioned with an ASG.
120+
* All `EC2NodeClass` objects must have the `spec.amiFamily` set to `Custom`.
121+
* `spec.instanceStorePolicy` configuration is not supported in `EC2NodeClass`.
122+
* `spec.kubelet`, `spec.taints` and `spec.labels` configuration are not supported in `EC2NodeClass`, but they can be configured in the `Cluster` or `InstanceGroup` spec.

0 commit comments

Comments
 (0)