[WIP] Initial change to add more KCP roles. - #18495
Open
cheftako wants to merge 1 commit into
Open
Conversation
k8s-ci-robot
added
do-not-merge/work-in-progress
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
k8s-ci-robot
added
area/nodeup
area/provider/aws
Contributor
|
@cheftako: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Contributor
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kubernetes-prow
Bot
added
area/kops-controller
area/provider/azure
Added helper functions for subroles. Code does not add new functionality. It is making the API changes in preperation for multiple sub-roles within a role. A few key test fixes. Added canonicalization of the Instance Group Sub Resources.
Contributor
|
@cheftako: The following tests failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
cheftako
commented
Jun 25, 2026
| // "subnets": // Should not matter | ||
| ig.Spec.Zones = []string{failureDomain} | ||
| ig.Spec.Role = "Node" // TODO: Support other roles? | ||
| ig.Spec.Role = kops.InstanceGroupSubRoleNode.Role() // TODO: Support other roles? |
Member
Author
There was a problem hiding this comment.
Would it be better to make Role() more of a constructor which took 1 or more subroles?
Member
There was a problem hiding this comment.
I like the type safety we have here. I wish we didn't have to pass .Role(), but I like this (so far, but I'm only one line in!)
cheftako
commented
Jun 25, 2026
| switch ig.Spec.Role { | ||
| case api.InstanceGroupRoleControlPlane: | ||
| switch { | ||
| case ig.Spec.Role.HasControlPlane(): |
Member
Author
There was a problem hiding this comment.
Would ContainsControlPlane() be a better name method name than HasControlPlane()?
Member
There was a problem hiding this comment.
We might end up getting more specific e.g. RunsKubeApiserver, RunsKubeScheduler, RunsEtcd. For now, I think it's fine, because it's internal (i.e. not part of our API)
cheftako
commented
Jun 25, 2026
| allRoles = append(allRoles, r.ToLowerString()) | ||
| // TODO: Can we GA the APIServerNodes feature flag? | ||
| // TODO: Do we need feature flag for the new roles and multi role support? | ||
| allRoles = append(allRoles, role.ToLowerString()) |
Member
Author
There was a problem hiding this comment.
With canonicalization else where, do we still need ToLowerString() ?
cheftako
commented
Jun 25, 2026
| switch g.Spec.Role { | ||
| case "": | ||
| switch { | ||
| case g.Spec.Role == "": |
Member
Author
There was a problem hiding this comment.
Should this be "unknown" (or both)?
cheftako
commented
Jun 25, 2026
|
|
||
| allRoles := make([]string, 0, len(kopsapi.AllInstanceGroupRoles)) | ||
| for _, r := range kopsapi.AllInstanceGroupRoles { | ||
| if r == kopsapi.InstanceGroupRoleAPIServer && !featureflag.APIServerNodes.Enabled() { |
Member
Author
There was a problem hiding this comment.
I think we need a similar feature flag for multiple/new control plane roles.
cheftako
commented
Jun 25, 2026
| // InstanceGroupRoleAPIServer is an API server role. | ||
| InstanceGroupRoleAPIServer InstanceGroupRole = "APIServer" | ||
| // InstanceGroupSubRoleControlPlane is a control-plane sub-role. | ||
| InstanceGroupSubRoleControlPlane InstanceGroupSubRole = "ControlPlane" |
Member
Author
There was a problem hiding this comment.
Would it make sense to just make the the Instance version lower case and have it be the canonical form? So far I have been resisting as the current mechanism should discourage == and != as they should generally break. Thus encouraging folks to use the HasX() type APIs.
Member
There was a problem hiding this comment.
This is modeled after an enum. We're making it into an enumSet. We don't want to change the values, as that will be visible in the API unless you're really careful. It's fine to change the go variable names, except that it is bloating the diff right now. So I might do InstanceGroupRoleControlPlane InstanceGroupSubRole = "ControlPlane"
justinsb
reviewed
Jun 25, 2026
|
|
||
| if role == kops.InstanceGroupRoleControlPlane { | ||
| if role.HasControlPlane() { | ||
| c.IsMaster = true |
Member
There was a problem hiding this comment.
We probably need to deprecate c.IsMaster, but agree that we should not do it in this PR
justinsb
reviewed
Jun 25, 2026
| } | ||
|
|
||
| if role == kops.InstanceGroupRoleControlPlane || role == kops.InstanceGroupRoleAPIServer { | ||
| if role.HasControlPlane() || role.HasAPIServer() { |
Member
There was a problem hiding this comment.
This is where it would be nice if this was just role.RunsKubeAPIServer, and that function returned true based if it has the ControlPlane or the APIServer role (and potentially other roles also).
justinsb
reviewed
Jun 25, 2026
| func (c *NodeupModelContext) InstallGVisorRuntime() bool { | ||
| return c.BootConfig != nil && | ||
| c.BootConfig.InstanceGroupRole == kops.InstanceGroupRoleNode && | ||
| c.BootConfig.InstanceGroupRole.HasNode() && |
Member
There was a problem hiding this comment.
Maybe we should start to transition away from HasNode to RunsWorkerPods or some similar concept
justinsb
reviewed
Jun 25, 2026
| case kops.InstanceGroupRoleNode: | ||
| case kops.InstanceGroupRoleBastion: | ||
| case kops.InstanceGroupRoleAPIServer: | ||
| case g.Spec.Role.HasNode(): |
Member
There was a problem hiding this comment.
So I feel like the existing code mixed validation that it is a valid enum, vs validation of e.g. the subnets for a control-plane node.
Shall we split the enum validation vs the more detailed validation for a control-plane node?
justinsb
reviewed
Jun 25, 2026
|
|
||
| if g.Spec.Role == kops.InstanceGroupRoleAPIServer { | ||
| case g.Spec.Role.HasEtcd(): | ||
| allErrs = append(allErrs, field.Forbidden(field.NewPath("spec", "role"), "Please implement ValidateEtcdInstanceGroup")) |
Member
There was a problem hiding this comment.
We may not need this, and it's going to break if e.g. ControlPlane role returns HasEtcd => true (which I think it should, because it does run etcd)
justinsb
reviewed
Jun 25, 2026
| warmPool := cluster.Spec.CloudProvider.AWS.WarmPool.ResolveDefaults(g) | ||
| if warmPool.MaxSize == nil || *warmPool.MaxSize != 0 { | ||
| if g.Spec.Role != kops.InstanceGroupRoleNode && g.Spec.Role != kops.InstanceGroupRoleAPIServer { | ||
| if !g.Spec.Role.HasNode() && !g.Spec.Role.HasAPIServer() { |
Member
There was a problem hiding this comment.
This is where we need to be careful. I think we're trying to exclude roles that run the "core control plane". I think it's fine to continue to check for specific Role values, though ideally we would better understand the reason and just check e.g. if RunsEtcd || RunsKubeControllerManager, or whatever we care about
justinsb
reviewed
Jun 25, 2026
| InstanceGroupRoleAPIServer InstanceGroupRole = "APIServer" | ||
| // InstanceGroupSubRoleControlPlane is a control-plane sub-role. | ||
| InstanceGroupSubRoleControlPlane InstanceGroupSubRole = "ControlPlane" | ||
| canocicalControlPlane InstanceGroupSubRole = "control-plane" |
justinsb
reviewed
Jun 25, 2026
| func canonical(igsr string) InstanceGroupSubRole { | ||
| input := strings.ToLower(string(igsr)) | ||
| switch input { | ||
| case "control-plane", "control-planes", "controlplane", "controlplanes": |
Member
There was a problem hiding this comment.
I think we verify these values on input. If we're thinking about this like a kubernetes API, maybe we accept "controlPlane" instead of "ControlPlane", but we probably don't need to accept "ControlPlanes" or even "control-plane"
justinsb
reviewed
Jun 25, 2026
| } | ||
|
|
||
| func (igsr InstanceGroupSubRole) Role() InstanceGroupRole { | ||
| return InstanceGroupRole(igsr) |
Member
There was a problem hiding this comment.
One thing we could do is to keep these all as type InstanceGroupRole, and defer supporting the comma-separated list until later. So what we're doing here is moving to the RunsEtcd / RunsKubeapiserver etc model, instead of looking directly at the enum values.
The diff should be much shorter as well!
justinsb
reviewed
Jun 25, 2026
|
|
||
| // IsControlPlane checks if instanceGroup is a control-plane node. | ||
| // IsControlPlane checks if instanceGroup has a control-plane sub-role. | ||
| func (g *InstanceGroup) IsControlPlane() bool { |
Member
There was a problem hiding this comment.
The problem with this method is that the answer is now often "it depends". So we might need to remove the method entirely, in favor of methods we can answer easily.
justinsb
reviewed
Jun 25, 2026
| func (b *FirewallModelBuilder) addHTTPSRules(c *fi.CloudupModelBuilderContext, sgMap map[string]*openstacktasks.SecurityGroup, useVIPACL bool) error { | ||
| masterName := b.SecurityGroupName(kops.InstanceGroupRoleControlPlane) | ||
| nodeName := b.SecurityGroupName(kops.InstanceGroupRoleNode) | ||
| masterName := b.SecurityGroupName(kops.InstanceGroupSubRoleControlPlane.Role()) |
Member
There was a problem hiding this comment.
I do like forcing a compilation error, but it is a lot of changes. It might be better to start by saying that these are still Roles, and just adding the "HasControlPlane" helpers etc in some strategic spots, and working to move to something more like what you have here over time.
Member
Author
|
/cc @hakman ref of the sort of change I'm headed towards. |
kubernetes-prow Bot
pushed a commit
that referenced
this pull request
Jun 27, 2026
Preperation for something like #18495. Moving away from direct comparison (== or !=) on IG role. Using helper methods such as HasNode() or HasControlPlane(). Also added a hack test so we don't backtrack. Should help prepare for supporting more control plane roles.
kubernetes-prow
Bot
added
the
needs-rebase
Contributor
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Paolo-Beci
added a commit
to Elemento-Modular-Cloud/kops
that referenced
this pull request
Jul 8, 2026
* Upgrade Go to 1.26.3 * Update dependencies Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * gomod: tidy and verify all modules Replace the hardcoded module lists in the `gomod` make target and verify-gomod.sh with dynamic discovery, so no module is left untidied or unverified. Hidden directories are excluded. * e2e: set the v3 runtime class path in apiserver templates * build: use gcr.io/distroless/static as base image Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * feat: add gVisor RuntimeClass support for containerd Add support for running workloads under gVisor (runsc) as a containerd runtime handler, following the existing NVIDIA GPU runtime pattern. gVisor is gated to Debian-family distributions only. The runsc apt package provides both runsc and containerd-shim-runsc-v1. Assisted by Opus 4.6 Signed-off-by: Arnaud Meukam <ameukam@gmail.com> * Use protobuf * hetzner: enable Cluster Autoscaler Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * chore(channels): promote alpha to stable Promotes kubernetes#18367 with latest k8s versions Signed-off-by: Moshe Vayner <moshe@vayner.me> * chore(networking): bump aws cni to version 1.21.2 Signed-off-by: Moshe Vayner <moshe@vayner.me> * test: hack/update-expected Signed-off-by: Moshe Vayner <moshe@vayner.me> * chore: don't try to use proto for bare-metal tooling * Release 1.36.0-alpha.1 (kubernetes#18413) Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * etcd: add ListenClientHTTPURLs field to EtcdManagerSpec * generated changes * Release notes for 1.36 (alpha) * chore: Add hashes for additional May releases * Add logs at v2 level to ensure we have slow pods data in logs * chore: upgrade containerd to v2.3.1 * ./hack/update-expected.sh * Upgrade kube-router to v2.10.0 * Disable strict external IP validation * azure: Use IMDS attested metadata document for node identity Replace the resourceID+vmID token with a PKCS7-signed attested metadata document from the Azure IMDS. The authenticator queries the attested document endpoint, and the verifier validates the PKCS7 signature chain, checks the nonce and expiration, then cross-verifies the signed vmId against the Azure API response. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * azure: Address review feedback on IMDS attestation - Thread context.Context through the IMDS query helpers and verifier client construction (http.NewRequestWithContext). CreateToken and the nodeidentity client use context.TODO() where no context is available. - Require the signer certificate to carry at least one issuer identifier (RawIssuer or AuthorityKeyId) before matching a fetched intermediate, so the per-field checks cannot degrade to accepting any CA. - Include the source URL when an intermediate certificate fails to parse. - Document the PKCS7 verification trust progression and clarify that validateFetchedIntermediateForSigner is a structural check only; trust is established by verifySignerCertChain. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * azure: Reflow comments to 100 columns Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * azure: Resolve intermediate CA chain across multiple AIA hops Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * aws: pin LBC selectors to fix in-place upgrades The helm+kustomize migration switched the controller Deployment and webhook Service selectors from {component, name} to the chart's {instance, name}. A Deployment's spec.selector is immutable, so upgrading from kops <1.36 has the apply rejected: the pod template is never updated, the webhook Service loses its endpoints, and the mutating webhooks (failurePolicy: Fail) time out, breaking target registration. Pin both selectors back to the historical {component, name} labels and add the component label to the pod template so it still satisfies the selector, keeping upgrades working without touching the immutable field. * cert-manager: set AWS_REGION on controller for Route53 dns-01 Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * cert-manager: upgrade to v1.19.5 * ./hack/update-expected.sh Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * Default omitted authorization to RBAC instead of AlwaysAllow A cluster spec applied with the top-level `authorization` field omitted (kops create -f / replace -f) defaulted to AlwaysAllow, while `kops create cluster` has always defaulted to RBAC. Align the v1alpha2 and v1alpha3 SetDefaults_ClusterSpec with the CLI: an omitted or empty authorization now defaults to RBAC. AlwaysAllow is still honored when set explicitly. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * cilium: allow disabling masquerade in ENI IPAM mode kOps forbids setting masquerade (disableMasquerade) when Cilium ENI IPAM is used, erroring with "Masquerade must be enabled when ENI IPAM is used". This blocks users who want Cilium's upstream no-masquerade behavior for ENI (e.g. private-topology clusters with NAT-gateway egress, or clusters using VPC endpoints), forcing them to patch the cilium-config ConfigMap after the fact, which races new nodes during rolling updates. Remove the validation so masquerade can be set in either direction for ENI. The default is intentionally left unchanged (masquerade stays on): flipping it off by default breaks pod egress to external endpoints (e.g. IRSA -> STS) on public-topology clusters whose pod ENI IPs have no public address, as shown by a red pull-kops-e2e-cni-cilium-eni run. Users who can route pod egress without masquerading opt in via masquerade=false (or via extraConfig). Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * channels: surface addon apply failures via a readiness probe Apply errors were only logged, so a manifest the apiserver rejects (e.g. an immutable Deployment selector) left the cluster silently broken. /readyz now reflects the last apply outcome; the pod is system-node-critical, so a NotReady fails `kops validate cluster` and halts the rolling update before workers roll. * etcd-manager: bump etcd to latest patches (3.5.31, 3.6.12) Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * etcd-manager: upgrade to v3.0.20260531 Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * ./hack/update-expected.sh Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * Build only linux/amd64 in kubernetes scalability presubmits * chore: downgrade containerd to v2.2.4 * ./hack/update-expected.sh * Skip the live kubelet version skew probe for Terraform golden tests Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * Ignore the default kubeconfig for hack/update-expected.sh Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * Allow feature gates to be set in scalability tests * dump: add --node-dump-timeout flag for per-node dump timeout * docs: add gVisor RuntimeClass support to 1.36 release notes Signed-off-by: Arnaud Meukam <ameukam@gmail.com> * Remove gVisor package config * Restrict gVisor runtime to worker instance groups gVisor (runsc) was previously installable on any instance group role. Restrict it to nodes with role Node: reject cluster/IG configs that enable gVisor on control plane, apiserver, or bastion roles, strip the gVisor config from non-worker nodeup configs, and only apply the gVisor node label and RuntimeClass addon when a worker has it enabled. Also harden nil handling for cluster.Spec.Containerd in nodeup config and bootstrapchannelbuilder. Update release notes and add tests across validation, nodeup config, gvisor builder, and instancegroup spec. * Enable misspell in golangci-lint Signed-off-by: Arnaud Meukam <ameukam@gmail.com> * build(deps): bump actions/checkout from 6.0.2 to 6.0.3 Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.2 to 6.0.3. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@de0fac2...df4cb1c) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * calico: add NFTablesMode setting Surface Calico's Felix NFTablesMode (Disabled, Enabled, Auto) as a field on CalicoNetworkingSpec and propagate it to the calico-node DaemonSet via FELIX_NFTABLESMODE. When left unset, the upstream Calico chart default applies, preserving existing behavior. On distributions where iptables is only present as a shim over nftables (e.g. RHEL10+, Rocky10+), routing Felix's data plane through iptables-nft / nft_compat has produced BGP session flapping and broken pod networking on GCE. This field lets clusters opt their Calico install into native nftables on those nodes. * flagbuilder: only shell-quote values for the joined string form * etcd-manager: switch to go-runner-based distroless image * ./hack/update-expected.sh Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * kube-proxy: assert buildPod command in unit test * Remove unused util/pkg/exec package * gVisor: add HasGVisor() helper function * Default node dump timeout to 5m in scalability run-test.sh * Support etcd 3.7.0-rc.0 and allow overriding etcd version in scalability scenario Signed-off-by: Jefftree <jeffrey.ying86@live.com> * Regenerate integration golden outputs for etcd 3.7.0-rc.0 Signed-off-by: Jefftree <jeffrey.ying86@live.com> * Fix GCE backend service ownership filtering during delete containsOnlyListedIGMs returned true for backend services with no backends (vacuous truth), causing listBackendServices to claim unrelated regional backend services. This cascaded into listHealthchecks selecting their health checks too. The delete loop then retried those foreign resources indefinitely. Return false when Backends is empty so only backend services with backends pointing at cluster IGMs are selected for deletion. Signed-off-by: Jathavedhan M <jathavedhan.m@ibm.com> * azure: Simplify IMDS attestation code * Filter randomized AWS zones by instance type availability * Release 1.36.0-beta.1 (kubernetes#18464) * Improve Google Storage Bucket support. Ensuring the image stored in gs:// can be retrieved. Switching from 'gsutil' to 'gcloud storage' where appropriate. Switching from 'gsutil' to 'gcloud buckets' where appropriate. go get cloud.google.com/go/storage ./hack/update-goimports.sh ./hack/verify-gomod.sh ./hack/update-expected.sh * gce: Reconcile HTTP health check changes on existing health checks Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * aws: Reconcile target group health check changes on existing target groups Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * Enable gocritic and gosec linters Changes are fixes from running 'make verify-golangci-lint' Signed-off-by: Arnaud Meukam <ameukam@gmail.com> * Added support for APIServer using LB Added support for APIServer without local Etcd Etcd is hosted on ControlPlane nodes. For GCP setup a LB and plumb it's address into /opt/kops/conf/kube_env.yaml Fix up /etc/hosts to point the etcd host to the LB address. Also added support for gs: storage buckets. Needed to use the hack/dev-build-gce.sh build. make gomod hack/update-expected.sh Fixed template for google storage copy & check Wired context up to make call stack more obvious Also provided better reuse of context object make gofmt make goimports Fixed "APIServer role forbidden on GCE with dns=None" test to reflect that use case should no longer generate an error Fixed unit test. * feat(linode): implement SSH key management and associated tasks Signed-off-by: Moshe Vayner <moshe@vayner.me> * Add serathius as scalability test owner * Report experiment variant into scalability run metadata * Asserted fixes from justinsb's feedback. One significant change was renaming EtcdLBAddresses as EtcdIPs. Cleanup k8s tf test file. * chore(channels): bump k8s versions in alpha channel Signed-off-by: Moshe Vayner <moshe@vayner.me> * Upgrade Karpenter to v1.13.0 Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * Replace deprecated karpenter.sh/provisioner-name with karpenter.sh/nodepool The karpenter.sh/provisioner-name label is a Karpenter v1alpha5 relic. Since v1beta1/v1, Karpenter applies karpenter.sh/nodepool to the nodes it provisions. Update the controller's anti-self-scheduling affinity to match Karpenter's own v1 label (also the upstream chart default), and stop setting the obsolete provisioner-name label on Karpenter-managed nodes. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * Remove obsolete node-role.kubernetes.io/master affinity term kOps supports Kubernetes 1.31+, where node-role.kubernetes.io/master no longer exists (removed in 1.24/1.25) and kOps only labels control-plane nodes with node-role.kubernetes.io/control-plane. The second nodeSelectorTerm keyed on the master label can never match, so the control-plane term alone suffices. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * Generate karpenter addon manifest with kustomize Adopt the same kustomize-based generation as the Azure CCM/CSI addons: the upstream chart is pulled via a kustomization.yaml helmChart, kops customizations (image, dnsPolicy, feature gates) are declarative patches, and the manifest is regenerated with regenerate.sh. This replaces the manual 'helm template' command plus hand-applied customizations, which required a 3-way merge to preserve on every version bump. Controller replicas are now fixed at 1, matching the other kustomize-generated addons. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * hack/update-expected.sh Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * channels: Add Kubernetes 1.36.0 support * Skip ImageVolume tests on COS 121 * gce: emit kops.k8s.io/instancegroup node label The GCE node identifier reads the instance-group name from the MIG instance template metadata but never adds it to the returned labels, so kops-controller never patches kops.k8s.io/instancegroup onto GCE nodes. The label has been missing since the cluster-api refactor in a7f1b4f. * Register Karpenter nodes with karpenter.sh/unregistered taint Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * hack/update-expected.sh Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * Add missing EC2 read permissions to Karpenter IAM policy Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * scaletest: report experiment variant after kubetest2 runs * nodeup: load ip_set module and disable firewalld on RHEL10+/Rocky10+ Two related fixes for Calico on the ForceNftables() distros (RHEL10+, Rocky10+, etc.). Load the ip_set kernel module alongside nf_tables and nf_conntrack. Calico's Felix unconditionally starts an ipsetsManager that shells out to "ipset list -name" during dataplane resync, even when NFTablesMode is Enabled. On RHEL10-family kernels ip_set is not auto-loaded, so the ipset call returns EINVAL and Felix panics in a tight loop, crashing calico-node and blocking cluster Up on every arm64 grid cell. Disable and mask firewalld via a new disableFirewalld step on FirewallBuilder, gated on Distribution.ForceNftables(). firewalld's default-reject filter_INPUT/filter_FORWARD policies and periodic-reload behavior conflict with the iptables/nftables rules CNIs install for pod and service traffic; Calico's own requirements doc and RKE2 both document that firewalld must be disabled on hosts running these CNIs. The disable/mask sequence is idempotent and a no-op where firewalld is not installed, so this is net-neutral on the cloud images that already strip firewalld (AWS RHEL/Rocky AMIs, Rocky GenericCloud) and net- positive on the GCE-optimized Rocky 10 image where firewalld ships active and breaks Calico BGP keepalives in BPF mode. * Adding flag to enable machine type for APIServer only. Added flag --api-server-size to be consistent with other machine type flags. Added doc on the flag reflecting my testing. Adding GCE test for APIServer only option. Fixed comment from previous PR. apiserver only DNS check for AWS comment is now correct. Removed k8s version flag from doc. make gen-cli-docs * Adding flag to enable machine type for APIServer only. * Adding GCE test for APIServer only option. * Fixing LB behavior when you have both APIServer and Control Plane. Initially the LB sent traffic to both. The DNS None is a new case. Now we only send traffic to the APIServer in this case. This protects the Control Plane nodes to do core controller work. Remove separate tests. Regenerated docs. * Remove namespace from DO ClusterRole * chore(channels): promote alpha to stable * chore(channels): bump node images * chore(channels): recommend kOps 1.35.1 for k8s 1.30-1.35, 1.34.3 for 1.29 * chore(channels): add arm64 GCE and Azure noble node images * ./hack/update-expected.sh * scale-test: bind etcd metrics to all interfaces * build(deps): bump actions/checkout from 6.0.3 to 7.0.0 Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.3 to 7.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@df4cb1c...9c091bb) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * e2e: test the PR's own channels, not master's In e2e, `kops create cluster --channel=alpha` reads the channel from the kops master branch, so a PR's edits to channels/alpha or channels/stable are never exercised by its own e2e jobs. When kops is built from the PR checkout, the deployer now rewrites --channel to a file:// path into that checkout's channels/ directory (defaulting to alpha when --channel is unset), so the build uses the PR's channels. Downloaded release/marker binaries don't match the checkout and keep using master's channels. * tests/e2e: refine externalTrafficPolicy=Local and hostNetwork skips The externalTrafficPolicy=Local source-IP-preservation tests only fail on Cilium (the client IP is SNATed to a pod IP instead of being preserved), tracked upstream in cilium/cilium#37613. Move the "implement NodePort and HealthCheckNodePort correctly when ExternalTrafficPolicy changes" skip into the Cilium block next to its sibling so other CNIs run the test. The hostNetwork "function for service endpoints" test was fixed in k8s 1.37 by kubernetes/kubernetes#139819 (it now reads spec.nodeName via the Downward API instead of os.Hostname()), so drop its skip gate from < 1.38 to < 1.37. Also clean up stale/incorrect issue references in the surrounding comments (wrong Azure issue, superseded hostname WIP PR, and the unrelated #129221). * Update CCM pods to tolerate all taints * ./hack/update-expected.sh * Allow setting missing slice elements from the command line Grow slices for explicit indexes while processing --set paths, so paths like cluster.spec.addons[0].manifest can create the first element. * tests/ai-conformance: install Gateway API CRDs through kOps addons Configure the scenario to install Gateway API CRDs via cluster.spec.addons, using the Gateway API version documented by Istio 1.29. * channels: apply direct manifests from spec.addons * Add managed Karpenter EC2NodeClass and NodePool Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * Switching from comparison on Role to helper. Preperation for something like kubernetes#18495. Moving away from direct comparison (== or !=) on IG role. Using helper methods such as HasNode() or HasControlPlane(). Also added a hack test so we don't backtrack. Should help prepare for supporting more control plane roles. * tests/e2e: skip implement-NodePort ETP=Local test on more CNIs The "Services should implement NodePort and HealthCheckNodePort correctly when ExternalTrafficPolicy changes" test was previously gated to Cilium only, but the e2e-kops-aws-cni-* periodic jobs show it also fails on flannel, kopeio and kube-router: the client source IP is SNATed to a pod IP instead of being preserved (kube-router instead times out reaching the local endpoint). It is the sole failure in those three jobs' latest runs. Move it out of the Cilium block into a condition covering cilium, flannel, kopeio and kube-router. amazon-vpc, calico and kindnet preserve the source IP and continue running the test. The sibling "externalTrafficPolicy=Local for type=NodePort" test passes on every non-Cilium CNI, so it stays gated to Cilium only. * chore(channels): pin Azure noble image to a deployable version Azure retired the pinned Ubuntu 24.04 daily images from the uksouth marketplace, so VMSS creation fails with PlatformImageNotFound and every Azure e2e job dies in the Up phase: The platform image 'Canonical:ubuntu-24_04-lts:server:24.04.202606120' is not available. `az vm image show` (the query path a deployment uses) confirms 24.04.202606120 (amd64) and 24.04.202606110 (arm64) are no longer available, while `az vm image list` still lists them from a stale catalog. 24.04.202606060 is the newest version that `az vm image show` confirms deployable for both server and server-arm64, so pin both arches to it. * tests/e2e: also skip implement-NodePort ETP=Local test on calico+GCE The "Services should implement NodePort and HealthCheckNodePort correctly when ExternalTrafficPolicy changes" test fails on calico on GCE but passes on calico on AWS. On GCE the VPC drops packets with arbitrary calico pod-CIDR source/dest addresses, so calico must IPIP-encapsulate inter-node pod traffic (routes go via tunl0). The IPIP/masquerade path rewrites the ETP=Local NodePort traffic's source to the node's tunnel address (a pod-CIDR IP) instead of preserving the client IP. On AWS kops disables the EC2 source/dest check, so calico routes pod traffic natively over the VPC (dev ens5, no encapsulation) and the source IP is preserved. Extend the skip to calico when the cloud provider is GCE. amazon-vpc and kindnet continue running the test on both clouds. * build(deps): bump actions/setup-go from 6.4.0 to 6.5.0 Bumps [actions/setup-go](https://github.com/actions/setup-go) from 6.4.0 to 6.5.0. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@4a36011...924ae3a) --- updated-dependencies: - dependency-name: actions/setup-go dependency-version: 6.5.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Add an experimental roles feature flag. No new functionality yet. Added 4 new role placholders, etcd, scheduler, ccm and kcm. Sets up the CLI API as well as the accessor functions. * azure: Scope nodes-to-API NSG rules to the NAT gateway public IP The AllowNodesToKubernetesAPI and AllowNodesToKopsController rules allowed any source, which bypassed the spec.api.access allowlist on port 443 and exposed kops-controller to the internet on clusters with a public API load balancer. Node traffic to the public frontend egresses through the NAT gateway, so its public IP is the only source these rules need. * fix: use JoinHostPost to tolerate IPv6 addresses * azure: Bump azuredisk-csi-driver to v1.34.4 The kops Azure e2e jobs (e.g. e2e-kops-azure-gossip-ha) flake on slow Azure disk attach/detach. The v1.34.x patch stream fixes that path: - kubernetes-sigs/azuredisk-csi-driver#3481 (v1.34.1): fix: handling dangling detaches in a better way - kubernetes-sigs/azuredisk-csi-driver#3547 (v1.34.3): fix: incorrect node attached on waitingForDetached call * ./hack/update-expected.sh * azure: Move cloud-controller-manager tolerations into the kustomize patch "Update CCM pods to tolerate all taints" hand-edited the generated k8s-1.31.yaml.template to set the cloud-controller-manager and cloud-node-manager tolerations to a single "operator: Exists". That override lived only in the template, not in any kustomize input, so regenerate.sh reverted it to the chart's default tolerations and would silently drop the override on the next regeneration. Move the override into the kustomization.yaml patches block so regenerate.sh reproduces it. The rendered template is unchanged. * azure: Bump azure-cloud-controller-manager to v1.36.2 cloud-provider-azure stopped publishing images to mcr.microsoft.com/oss/kubernetes after v1.34.3 and now publishes to mcr.microsoft.com/oss/v2/kubernetes, which the upstream Helm chart selects for Kubernetes 1.32 and newer. Switch the cloud-controller-manager and cloud-node-manager image defaults to the oss/v2 path so kOps can track current releases, and bump both from v1.34.3 to v1.36.2. Also bump the pinned Helm chart from 1.34.5 to 1.36.0 to keep the chart minor aligned with the image minor. The rendered manifest is unchanged: the chart's Deployment and DaemonSet templates are identical between the two versions, and kOps patches the image fields out with the {{ .ExternalCloudControllerManager.* }} placeholders. Notable upstream changes since v1.34.3: - New service.beta.kubernetes.io/azure-disable-load-balancer-nsg-rule annotation to skip CCM-managed LoadBalancer NSG rules. - External LoadBalancer Services with an invalid or non-public pinned IP now fail fast instead of listing Public IPs. - Multiple standard load balancer fixes: backend pool IP deduplication, serialized backend pool updates, and IP sharing across services. - Network-isolated clusters always use the managed identity credential. - ACR credential provider adds KSA-based authentication. Release notes: https://github.com/kubernetes-sigs/cloud-provider-azure/releases * ./hack/update-expected.sh * chore: update syntax of CLAUDE.md The @ syntax is more readily followed by claude. * azure: Grant control-plane VMSS Contributor instead of Owner The built-in Owner role includes Microsoft.Authorization/roleAssignments/write, a privilege-escalation path reachable by any pod via the instance metadata endpoint. No cluster component uses Microsoft.Authorization; Contributor covers everything the control plane needs. Existing clusters keep the old Owner role assignment until it is removed manually or the cluster is deleted; the terraform target prunes it on apply. * coredns: Honor node taints in hostname topologySpreadConstraint The hostname topologySpreadConstraint uses whenUnsatisfiable: DoNotSchedule, but CoreDNS autoscales via the cluster-proportional-autoscaler. On clusters with a single schedulable worker (e.g. one control-plane + one worker), the tainted control-plane node was counted as an empty topology domain, so the second CoreDNS replica could never satisfy maxSkew and stayed Pending. Setting nodeTaintsPolicy: Honor excludes nodes with untolerated taints (control-plane, cordoned/draining) from the skew calculation, keeping the hard spread across workers while allowing the replicas to co-locate when there is only one eligible node. * ./hack/update-expected.sh * Expose GCP project as PROJECT env for ClusterLoader2 on GCE ClusterLoader2's managed Prometheus client (clusterloader2/pkg/prometheus/clients/gcp_managed.go) reads the GCP project from the PROJECT env var to build the Cloud Monitoring query URL. The kubetest2-kops deployer only exported GCP_PROJECT, so for scalability jobs that enable the "Quotas total usage" measurement (via preset-e2e-scalability-common's CL2_ENABLE_QUOTAS_USAGE_MEASUREMENT) the query URL had an empty project and Cloud Monitoring rejected it with HTTP 400 INVALID_ARGUMENT, failing the load test. Also export PROJECT alongside GCP_PROJECT so the measurement can query the correct project. This is set from d.GCPProject, which is populated even when the project is acquired from boskos. * hetzner: fix Cluster Autoscaler node group membership hetzner: improve Cluster Autoscaler integration The Cluster Autoscaler Hetzner provider keys node group membership, initial target size and Nodes() entirely off the hcloud/node-group server label, which kOps never set. As a result kOps-created servers were invisible to the autoscaler. - Label Node-role servers with hcloud/node-group at creation. Servers from earlier kOps versions get the label when they are replaced. - Treat minSize as a floor: servers beyond Count are no longer marked needs-update, so autoscaler scale-ups survive kops update cluster. Shrink with 'kops delete instance' or autoscaler scale-down. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * tests/e2e: skip implement-NodePort ETP=Local test on calico+Azure The "Services should implement NodePort and HealthCheckNodePort correctly when ExternalTrafficPolicy changes" test fails on calico on Azure for the same reason it does on GCE: the underlay cannot route the pod CIDR, so calico must encapsulate inter-node pod traffic (VXLAN vxlan.calico on Azure, IPIP tunl0 on GCE). On the node-local ExternalTrafficPolicy=Local NodePort short-circuit path the masquerade then rewrites the client source IP to the node's tunnel address (a pod-CIDR IP) instead of preserving it, so the test fails on every run once the cluster comes up. Calico preserves the source IP only on AWS, where kOps disables the source/dest check and routes pod traffic natively. Extend the calico skip from gce to also cover azure. * Update cluster-autoscaler to v1.36.0 Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> * tests: enable cluster-autoscaler in the ha_gce integration test The nodes instance group in this scenario spans two zones, so kOps creates two zonal InstanceGroupManagers, but only the first zone's MIG is registered with cluster-autoscaler. The expected output captures this current behavior. * gce: register all zonal MIGs of a multi-zone instance group with cluster-autoscaler For a GCE instance group spanning multiple zones, kOps creates one zonal InstanceGroupManager per zone, but only registered the first zone's MIG with cluster-autoscaler, carrying the full min/max of the instance group. The MIGs in the remaining zones were invisible to the autoscaler and were never scaled, so scale-ups could get stuck in a single zone (for example when that zone has no spot capacity). Register each zonal MIG as its own cluster-autoscaler node group, splitting the instance group's min/max sizes across zones with the same algorithm used for the MIG target sizes. Single-zone instance groups render exactly as before. * tests: update expected output for ha_gce * Update Go to 1.26.5 and bump golang.org/x modules * clusterapi: fix build by making gen.go a library package * pkg/assets: keep container registry clients out of runtime binaries * rebase with upstream lug 2026 * update: vendor * feat: logic up to the region choose func * feat: added various integrations * feat: daemons api endpoints and other improvements * various errors resolution * feat: working prototype up to store phase * working create config generation * rebase kops lug 2026 * feat: servergroup initial implementation * fix: compatibility issues * fix: copute instance creation * feat: volume support * fix: compatibility with elemento lib * fix: bugs * fix: remove verify token between nodes for development purpose * fix: minor * fix: nodeup auto script * chore: changed module repo from tesi-paolobeci to ecloud-go * fix: version elemento * fix: minor * feat: custom ssh key support * feat: limited task execution retries on error * fix: remote ecloud package on go.mod * fix: release secret * fix: deactivate useless gh actions * fix: ecloud version * fix: minor --------- Signed-off-by: Ciprian Hacman <ciprian@hakman.dev> Signed-off-by: Arnaud Meukam <ameukam@gmail.com> Signed-off-by: Moshe Vayner <moshe@vayner.me> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Jefftree <jeffrey.ying86@live.com> Signed-off-by: Jathavedhan M <jathavedhan.m@ibm.com> Co-authored-by: Ciprian Hacman <ciprian@hakman.dev> Co-authored-by: Kubernetes Prow Robot <20407524+k8s-ci-robot@users.noreply.github.com> Co-authored-by: Arnaud Meukam <ameukam@gmail.com> Co-authored-by: Marek Siarkowicz <serathius@users.noreply.github.com> Co-authored-by: Moshe Vayner <moshe@vayner.me> Co-authored-by: justinsb <justinsb@google.com> Co-authored-by: Jefftree <jeffrey.ying86@live.com> Co-authored-by: Harish K <hakuna@amazon.com> Co-authored-by: Peter Rifel <pgrifel@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Jathavedhan M <jathavedhan.m@ibm.com> Co-authored-by: Walter Fender <wfender@google.com> Co-authored-by: kubernetes-prow[bot] <181008794+kubernetes-prow[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Looking to add Etcd, Scheduler , Cloud Controller Manager and Kube Controller Manager to the list of KCP roles.
(Which is currently API Server) Making the API changes to allow this to happen. No actual functionality additions have been made as part of this change.