Skip to content

WIP azure: serve cluster assets from a kOps-managed container registry - #18603

Draft
hakman wants to merge 7 commits into
kubernetes:masterfrom
hakman:azure-oci-asset-registry
Draft

WIP azure: serve cluster assets from a kOps-managed container registry#18603
hakman wants to merge 7 commits into
kubernetes:masterfrom
hakman:azure-oci-asset-registry

Conversation

@hakman

@hakman hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member

This adds support for serving all of a cluster's assets, both files (nodeup, kubelet, containerd, CNI) and container images, from a dedicated per-cluster Azure Container Registry that kOps creates and manages.

Motivation

Nodes currently download file assets from public mirrors and pull images from public registries. A dedicated per-cluster registry removes the public egress dependency, keeps all pulls same-region (free on Azure), and gives content-addressed integrity: OCI blob digests are the sha256 of the content, so the hash kOps already embeds in the bootstrap script doubles as the download address, with no manifest parsing needed on the node.

Usage

spec:
  assets:
    managed: true

With only managed: true, the registry name is derived from the subscription ID and the cluster name (registry names are global, and cluster names such as my.k8s are not unique across users). The derived fileRepository and containerRegistry values are visible with kops get cluster -o yaml --full, and both can also be set explicitly. An existing registry can be referenced without managed by setting an oci:// fileRepository directly.

Implementation

  • New assets.managed field (*bool) and support for the oci:// scheme in assets.fileRepository, validated as Azure-only for now.
  • A ContainerRegistry task creates the registry (Basic SKU, admin user disabled, no anonymous pull) in the cluster's resource group, so kops delete cluster cleans it up; the registry is also enumerated for the deletion preview. All instance groups get an AcrPull role assignment scoped to the registry.
  • kops update cluster pushes the assets via a RegistryAssets task ordered after registry creation. Files are pushed as single-layer OCI artifacts whose blob digest equals the file hash; images reuse the existing assetcopy machinery with an Entra ID keychain. The push function is injected by the CLI so that runtime binaries (nodeup, kops-controller) do not link the registry client libraries. kops get assets --copy also supports oci:// targets, which is the flow for the terraform target.
  • The bootstrap script and nodeup's asset store download blobs by digest, authenticating with the instance's managed identity (IMDS token, ACR token exchange). Pod and sandbox images are pulled through the acr-credential-provider kubelet plugin from cloud-provider-azure.

Notes

  • The registry endpoint is public DNS with authentication required for all content; Basic SKU does not support Private Link. Private endpoints would come with a future SKU option.
  • The registry is created with the classic (Unsecure) domain name label, where the resource name equals the DNS label. Domain name label scopes can be considered later, since the login server is only known after creation.
  • AWS (ECR) and GCP (Artifact Registry) support is planned as follow-up; the validation gate to Azure is the only thing to relax on the API side.

@kubernetes-prow

kubernetes-prow Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@kubernetes-prow kubernetes-prow Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 17, 2026
@kubernetes-prow

kubernetes-prow Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rifelpet for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow
kubernetes-prow Bot requested review from olemarkus and zetaab July 17, 2026 05:03
@hakman
hakman requested review from justinsb and rifelpet and removed request for olemarkus and zetaab July 17, 2026 05:03
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch from c46779d to 3f36f95 Compare July 17, 2026 05:23
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch 2 times, most recently from 3e07459 to 9600a69 Compare July 17, 2026 05:47
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch from 9600a69 to 586a130 Compare July 17, 2026 06:12
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch from 586a130 to b159da0 Compare July 17, 2026 07:25
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

1 similar comment
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch from b159da0 to 566183f Compare July 17, 2026 08:12
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

1 similar comment
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch from 566183f to 1895888 Compare July 17, 2026 08:39
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch from 1895888 to 71bb57a Compare July 17, 2026 09:10
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

1 similar comment
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch from 71bb57a to 53e298e Compare July 17, 2026 12:09
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch from 53e298e to 640908c Compare July 17, 2026 12:29
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch from 640908c to 00bb70a Compare July 17, 2026 13:26
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

hakman added 6 commits July 17, 2026 16:51
@hakman
azure: add assets.managed API field
2eac5de 
An oci:// assets.fileRepository serves file assets from an OCI registry.
With assets.managed enabled, kOps creates and manages the Azure Container
Registry holding the cluster's file and image assets, as part of the
cluster's cloud resources.

When fileRepository and containerRegistry are not set, they default to a
registry name derived from the subscription ID and the cluster name;
registry names are global and cluster names alone are not unique. The
AssetBuilder is created before defaulting runs, so the defaulted
locations are propagated to it for asset remapping.

Asset paths are sanitized to valid OCI repository names, and the
sideloaded kops images of development builds are not remapped.
@hakman
azure: create and clean up a per-cluster container registry for assets
afd4715 
Adds the ContainerRegistry task (Basic SKU, in the cluster's resource
group), an AcrPull role assignment for all instance groups, the
RegistryAssets task that pushes assets during kops update cluster, and
delete-cluster enumeration of the registry.

The full list of assets is only known once all the bootstrap configs
and the addon manifests have been built, so the push runs after the
BootstrapScript and AddonManifest tasks; the RunsAfterAddonManifests
marker excludes the push from the addon tasks' wait-for-everything
dependencies. A newly created registry can reject data-plane requests
for several minutes while permissions propagate, so the push retries
failures for up to 15 minutes. Deleting a role assignment whose scope
is already deleted is treated as success.
@hakman
assets: push file and image assets to an oci:// fileRepository
d9cc248 
File assets are pushed as single-layer OCI artifacts whose blob digest is
the file's sha256 hash, so nodes can download blobs directly by hash.
Azure Container Registries are authenticated by exchanging an Entra ID
token for a registry refresh token; kops update cluster pushes assets
via the injected assetcopy.Copy so that only the CLI links the registry
client libraries.
@hakman
azure: download assets from the OCI registry on nodes
d1c3d02 
The bootstrap script and nodeup's asset store fetch OCI blobs by digest,
authenticating with the instance's managed identity. Pod images are
pulled through the acr-credential-provider kubelet plugin. containerd
pulls the sandbox image itself, without kubelet's image credential
providers, so nodeup pre-pulls it with the instance's managed identity.
@hakman
docs: document oci:// fileRepository and assets.managed
f76da87
@hakman
hack: enable assets.managed in the azure e2e jobs
4a8e680 
Do not merge; drop before merging.
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@hakman
hakman force-pushed the azure-oci-asset-registry branch from 00bb70a to 4a8e680 Compare July 17, 2026 13:51
@hakman

hakman commented Jul 17, 2026

Copy link
Copy Markdown
Member Author

/test pull-kops-e2e-azure-cni-cilium

@kubernetes-prow

kubernetes-prow Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@hakman: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kops-e2e-azure-cni-cilium 4a8e680 link true /test pull-kops-e2e-azure-cni-cilium

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@kubernetes-prow

kubernetes-prow Bot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 18, 2026
@hakman hakman mentioned this pull request Jul 18, 2026
Open
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@justinsb justinsb Awaiting requested review from justinsb
@rifelpet rifelpet Awaiting requested review from rifelpet

Assignees

No one assigned

Labels

area/api area/documentation area/nodeup area/provider/azure Issues or PRs related to azure provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@hakman