operator: update setup-envtest — kubebuilder-tools GCS bucket deprecated, all envtest tests fail in CI

[sbckr]

Problem

The operator.build-and-test workflow currently fails on every PR. The setup-envtest binary pinned in the Makefile attempts to download kubebuilder test asset binaries (kube-apiserver, etcd) from the legacy GCS bucket storage.googleapis.com/kubebuilder-tools, which now returns 401 Unauthorized. The bucket was deprecated as part of a Kubernetes infrastructure policy change (see kubernetes-sigs/kubebuilder discussion #4082).

The effect in CI:

KUBEBUILDER_ASSETS=""

All controller tests that require envtest to spin up a local kube-apiserver then fail with:

exec: "etcd": executable file not found in $PATH

This affects every Action run / PR to this repository.

Root Cause

klyshko-operator/Makefile installs setup-envtest at a specific pseudo-version:

ENVTEST_VERSION = v0.0.0-20230216140739-c98506dc3b8e

This commit is from February 2023 and still uses the deprecated GCS download path. The tool has no fallback.

Fix

The GCS issue was resolved in kubernetes-sigs/controller-runtime#2811, which landed in controller-runtime v0.19.0 (May 2024). That release switched setup-envtest's download source from the defunct GCS bucket to a new index hosted at:

https://raw.githubusercontent.com/kubernetes-sigs/controller-tools/HEAD/envtest-releases.yaml

Upgrade Requirements

Upgrading to a setup-envtest version that contains the fix is not a drop-in change. It requires:

Component	Current	Required
Go	1.16 (declared in go.mod)	≥ 1.22
controller-runtime	v0.9.2	≥ v0.19.0
Kubernetes client libraries	v0.22.x	v0.31.x

controller-runtime v0.19.0 is the first version containing the GCS fix, and its tools/setup-envtest/go.mod declares go 1.22.0. The go-get-tool Makefile macro used to install setup-envtest also relies on go get for binary installation, which was deprecated for this use in Go 1.17 — so the Makefile tooling will need updating regardless.

The controller-runtime v0.9.2 → v0.19.x upgrade is a major dependency bump and will require validation of the operator controllers against the updated API surface.

Available Binary Sources

Kubernetes 1.21 is not present in the new index — the oldest entry is v1.23.5. There is no official public mirror for the 1.21 binaries; they existed only on the now-defunct GCS bucket.

From v1.23.5 onwards, binaries are hosted directly on the kubernetes-sigs/controller-tools GitHub Releases page as envtest-v<k8s-version>-<os>-<arch>.tar.gz assets. Both kube-apiserver (from kubernetes/kubernetes) and etcd (from etcd-io/etcd) are Apache 2.0 licensed, and the controller-tools project already redistributes them under that licence — so fetching directly from those releases in CI requires no re-hosting.

Potential Quick-Fix Option (no Go or dependency upgrade required)

A targeted CI-only fix restores working test coverage immediately, without touching go.mod, Go version, or controller-runtime:

1. Bump ENVTEST_K8S_VERSION in the Makefile from 1.21 to 1.23 (the oldest version available in the new index).

2. In the CI workflow, bypass setup-envtest and download the tarball directly from controller-tools GitHub Releases, then set KUBEBUILDER_ASSETS before make test:

- name: Download envtest binaries
  run: |
    curl -sL https://github.com/kubernetes-sigs/controller-tools/releases/download/envtest-v1.23.5/envtest-v1.23.5-linux-amd64.tar.gz \
      | tar -xz -C /tmp/envtest-bins
    echo "KUBEBUILDER_ASSETS=/tmp/envtest-bins" >> $GITHUB_ENV

3. Guard the Makefile envtest target to skip the setup-envtest download step when KUBEBUILDER_ASSETS is already set (a ifndef KUBEBUILDER_ASSETS check around the download call is sufficient).

This requires only a Makefile variable bump and a small workflow addition. The full Go 1.22 / controller-runtime v0.19 upgrade can follow as a separate, properly validated infrastructure PR.

Compatibility note (Go 1.16 / controller-runtime v0.9.2 with k8s 1.23 binaries): --insecure-port was not removed from kube-apiserver until 1.24 (deprecated in 1.20, still accepted as 0 in 1.22 and 1.23). Additionally, CR v0.9.2's envtest package includes a discoverFlags() mechanism that probes the binary via --help at startup and strips any unrecognised flags — providing a degree of forward compatibility. Using 1.23 binaries with CR v0.9.2 on Go 1.16 is therefore expected to work. This should be validated when implementing the fix.

Acceptance Criteria

• operator.build-and-test workflow passes on a clean runner
• Quick-fix path: ENVTEST_K8S_VERSION bumped to 1.23, workflow fetches binaries directly from controller-tools GitHub Releases, no Go/dependency changes required
• Long-term path: full Go 1.22 + controller-runtime v0.19 upgrade validated in a dedicated PR