gce: add support karpenter-managed Instance groups - #18628
Draft
hakman wants to merge 6 commits into
Draft
Conversation
Contributor
|
Skipping CI for Draft Pull Request. |
kubernetes-prow
Bot
added
do-not-merge/work-in-progress
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kubernetes-prow
Bot
added
area/api
area/documentation
area/provider/gcp
hakman
force-pushed
the
karpenter-gce
branch
2 times, most recently
from
July 26, 2026 19:19
4164ad2 to
c71f384
Compare
GCE label keys and values cannot contain "/", "." or ":", so rendering cluster-autoscaler scale-from-zero taint hints as instance template labels makes 'kops update cluster' fail for any GCE instance group that sets spec.taints (for example a taint key like karpenter.sh/unregistered). On GCE the cluster autoscaler reads scale-from-zero taints from AUTOSCALER_ENV_VARS in the instance template's kube-env metadata, which kOps already emits, so these labels were never consumed there. Skip them on GCE, matching the existing handling of node label hints.
…ud groups) Allow spec.manager=Karpenter on GCE instance groups: - validation: accept GCE alongside AWS; keep the AMI selector checks and the IRSA requirement AWS-only (on GCE the controller uses the instance service account). - gcemodel: for Karpenter instance groups, build only the bootstrap script tasks (which publish igconfig/node/<ig>/nodeupscript.sh and nodeupconfig.yaml to the state store) and skip the InstanceTemplate and InstanceGroupManager, mirroring the AWS model. - populate spec: no default machine type for GCE Karpenter instance groups; without one the generated NodePool carries no instance-type requirement. - components: default cluster.spec.karpenter.image to the karpenter-provider-gcp controller image on GCE. - GetCloudGroups: synthesize a cloud instance group per Karpenter instance group, discovering instances by the kOps ownership labels the generated GCENodeClass applies, so 'kops validate cluster' and rolling-update see these nodes. The karpenter.sh/unregistered:NoExecute registration taint and the state-store script publication were already cloud-agnostic.
The GCE node identifier required MIG membership (via the created-by
metadata) to derive the owning instance group, so nodes launched
directly by Karpenter never received their role label or instance
group label ('cannot find owner for instance' in kops-controller logs).
When created-by is absent, derive ownership from the instance's own
kops-k8s-io-instance-group-name metadata after checking the exact
cluster-name metadata, mirroring the trust anchor already used by the
bootstrap TPM verifier: both values are read through the Compute API
and can only be changed with Compute API write permissions, which
nodes are not granted. MIG-managed instances keep the existing
membership-verified path.
…NodePool On GCE clusters with spec.karpenter.enabled, the karpenter.sh addon now deploys karpenter-provider-gcp in self-hosted mode (PROVISION_MODE, PROJECT_ID, CLUSTER_LOCATION, CLUSTER_NAME) on the control plane, using Application Default Credentials from the instance metadata server. One GCENodeClass and one NodePool are generated per Karpenter instance group, regenerated on every kops update cluster: - imageSelectorTerms from the instance group image; - boot disk from rootVolume settings with the MIG template defaults; - kubeletConfiguration.maxPods matching the kubelet (default 110); - the kOps ownership labels the MIG instance template would carry; - kops-k8s-io-instance-group-name metadata (bootstrap and node identity ownership), the node role network tag, the node service account, vTPM enabled (required by the bootstrap TPM verifier); - the instance group's nodeup script from the state store as startupScript, so a NodeupConfig change rolls nodes via drift. The addon manifest is generated from the karpenter-provider-gcp Helm chart via kustomize, mirroring the AWS pipeline; generated GCENodeClass objects are pruned when their instance group is removed. Self-hosted mode requires the karpenter-provider-gcp release containing cloudpilot-ai/karpenter-provider-gcp#511; the manifest was generated from that chart source and the pinned versions should be bumped to the first release that includes it.
…cumentation TestMinimalGossipGCE's fixture gains spec.karpenter.enabled and two Karpenter instance groups: one with a machine type and static capacity, one without a machine type (dynamic, custom maxPods), reusing the existing test rather than growing the GCE suite.
Generated with ./hack/update-expected.sh: - gossip-gce now includes the karpenter.sh addon and the generated GCENodeClass/NodePool objects; - ha_gce no longer renders the cluster-autoscaler taint hint as an instance template label.
kubernetes-prow
Bot
added
the
needs-rebase
Contributor
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds support for
spec.manager: Karpenteron GCE node InstanceGroups, mirroring the existing AWS integration. It deploys karpenter-provider-gcp in its new self-hosted provisioning mode (cloudpilot-ai/karpenter-provider-gcp#540), where nodes bootstrap through the kOps nodeup script and the controller uses only the Compute API, with no GKE dependency.For a Karpenter-managed InstanceGroup, kOps no longer creates an InstanceTemplate or MIG. Instead it publishes the IG's nodeup script to the state store and generates one
GCENodeClassand oneNodePoolper InstanceGroup, delivered by thekarpenter.shaddon and pruned when the IG is removed. The generatedGCENodeClasscarries the IG image, a boot disk from the rootVolume settings,kubeletConfiguration.maxPodsmatching the kubelet, the kOps ownership labels and instance group metadata, the node role network tag, the node service account, a Shielded VM config with vTPM enabled (required by the kops-controller TPM node authorization), and the nodeup script asstartupScript. Akops update clusterthat changes the IG's nodeup configuration updates thestartupScript, and Karpenter then rolls that NodeClass's nodes through drift, matching the AWSuserDatabehavior.