Serve cluster assets from OCI registries

[hakman]

/kind feature

Motivation

Nodes currently download file assets (nodeup, kubelet, containerd, CNI) and pull images from public mirrors and registries. Serving all of a cluster's assets from a single registry removes the dependency on public upstream sources, keeps downloads same-region when the registry is in the cluster's region, and enables clusters whose nodes cannot reach the public internet (the remaining external dependencies are the cloud's own endpoints: registry, metadata service, state store).

OCI registries are a natural fit for file assets because blob digests are the sha256 of the content: the hash kOps already embeds in the bootstrap script doubles as the download address (/v2/<repository>/blobs/sha256:<hash>). Nodes need no OCI client and no manifest parsing: the bootstrap script fetches the nodeup binary with curl, and nodeup fetches each remaining asset with a single HTTP GET after a short per-registry token exchange, with the existing hash validation verifying every download independently.

Target user experience, in increasing order of convenience:

# Step 1: any registry that allows anonymous pulls, on any cloud provider
spec:
  assets:
    fileRepository: oci://registry.example.com/assets

# Step 2: a pre-created private cloud registry, pulled with the instance identity
spec:
  assets:
    fileRepository: oci://myregistry.azurecr.io/assets

# Step 3: kOps creates and manages the registry; locations are defaulted
spec:
  assets:
    managed: true

Why no OCI client on nodes

Since kOps controls the push format, pulling an asset reduces to one authenticated GET of a well-known URL. A registry client (go-containerregistry, ORAS) would only be needed to resolve tags through manifests, and would add a large vendored dependency tree plus the per-cloud auth stacks to nodeup, a binary every node downloads at boot. The bootstrap script could not use a Go library either way, so it would still need the curl token flows. Instead, the per-cloud token exchanges are a few hundred lines of plain HTTP in Go, mirrored by short curl flows in the bootstrap script; in release clusters the script downloads exactly one artifact, the nodeup binary, and nodeup fetches the rest.

Plan

Each step delivers standalone user value; later steps build on the earlier ones.

1. oci:// fileRepository foundation (anonymous pulls, all cloud providers)

• 1a. kops get assets --copy pushes each file asset as a single-layer image whose blob digest is the file's hash, tagged with that hash; pushing authenticates with the local docker credentials, so any registry docker login can reach works.
• 1b. Nodes pull anonymously by digest; registries that require a token even for public pulls are handled via the anonymous token flow advertised in the WWW-Authenticate challenge (tested against Docker Hub and ghcr.io).

2. Private cloud registries (registry pre-created by the user)

Like state store buckets, the registry must already exist, and the instances must be granted pull access out of band. kOps pushes the assets and the nodes authenticate with their instance identity. The auth mode is inferred from the registry host (*.azurecr.io on Azure, *.pkg.dev on GCE, *.dkr.ecr.*.amazonaws.com on AWS, anonymous elsewhere), and each provider's registry is validated for use on its own cloud only.

• 2a. Azure: Container Registry. The bootstrap script and nodeup exchange an IMDS managed-identity token for a registry refresh token, then a pull-scoped access token.
• 2b. GCP: Artifact Registry. The metadata-server access token for the instance's service account is accepted directly as a bearer token. Kubelet image pulls already work today via containerRegistry and the existing gcp-credential-provider.
• 2c. AWS: Amazon ECR. Nodes obtain an ECR authorization token (accepted as HTTP basic auth) with the instance role credentials: nodeup signs the GetAuthorizationToken call with the vendored SigV4 signer, and the bootstrap script signs it with curl --aws-sigv4¹. Kubelet image pulls already work today via containerRegistry and the existing ecr-credential-provider.
• 2d. Azure: kubelet image pulls from the private registry, with the acr-credential-provider kubelet plugin from cloud-provider-azure.
• 2e. All providers: credentialed pre-pull of the sandbox image. containerd pulls the sandbox image itself, without kubelet's credential providers, so a private containerRegistry needs a pre-pull with the instance identity on every cloud, not just Azure.

3. kOps-managed registries: assets.managed: true

kOps creates the per-cluster registry resources in the cluster's cloud account, grants the instances pull access, pushes file and image assets during kops update cluster (ordered after the registry creation and after the full asset list is known; with the terraform target the push instead runs via kops get assets --copy after terraform apply), enumerates the resources in the deletion preview, and deletes them with the cluster.

• 3a. Azure: WIP azure: serve cluster assets from a kOps-managed container registry #18603. Creates a Container Registry (Basic SKU, admin user disabled) with AcrPull role assignments per VM scale set, and terraform support for the resources. Registry names are globally unique, so when fileRepository and containerRegistry are unset they default to a name derived from the subscription and cluster name.
• 3b. GCP: creates an Artifact Registry repository in the project (repository names are project-scoped, so no global-name derivation is needed).
• 3c. AWS: manages ECR repositories inside the account's existing registry (needs a repository-creation design, see open questions).

4. Bonus: authenticated object-store downloads in the bootstrap script

Support downloading nodeup itself from private buckets with curl, giving a private fileRepository without an OCI registry. nodeup can already read these stores natively; the gap is only the bootstrap script's download of the nodeup binary.

• GCS via a metadata-server bearer token (verified working)
• Azure Blob via an IMDS token for the storage resource
• S3: needs its own investigation; curl --aws-sigv4 proved unreliable for S3 requests across the distro fleet in testing (unlike the ECR API call), and presigned URLs are ruled out because nodes boot long after kops update cluster (SigV4 presigning caps at 7 days)

Open questions

• ECR repositories in managed mode: ECR does not create repositories on direct push, and the feature produces many repository names: image remapping flattens image paths into single-component names (coredns-coredns), while each file asset maps to its own repository path under the fileRepository prefix. The full name set is only known once the asset list is collected. Create the repositories from the collected list at push time, evaluate ECR repository creation templates (create-on-push), or restructure the file layout into a single repository (files are addressed by digest tag, so one repository could suffice for them)?

Out of scope

• Pruning: assets accumulate across upgrades and nothing is deleted until the cluster is. Storage is billed, not capped (ACR Basic includes 10 GiB), and years of upgrades cost little, so garbage collection is not worth the complexity for now.
• Private endpoints: ECR interface endpoints and Private Google Access work with these flows without kOps changes; ACR Basic does not support Private Link, and a SKU option can be added later.
• Authenticated third-party registries (e.g. Harbor with basic auth): nodes have no credential distribution story; only anonymous third-party registries are supported.
• Sovereign and China cloud partitions (.azurecr.us/.cn, .amazonaws.com.cn): the host gates and token audiences cover the public partitions only.

Footnotes

1. curl --aws-sigv4 signs the ECR call correctly from curl 7.86.0 on: Debian 12, Ubuntu 24.04, Amazon Linux 2023, and Flatcar qualify. ↩

[hakman]

/assign
CC @justinsb @rifelpet @ameukam