Skip to content

fix: make the kustomize manifests deployable and consistent - #2835

Open
AlinsRan wants to merge 4 commits into
apache:masterfrom
AlinsRan:chore/refresh-kustomize-manifests
Open

fix: make the kustomize manifests deployable and consistent#2835
AlinsRan wants to merge 4 commits into
apache:masterfrom
AlinsRan:chore/refresh-kustomize-manifests

Conversation

@AlinsRan

@AlinsRan AlinsRan commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Description

kustomize build config/default currently produces a deployment that cannot work. This PR fixes the manifests. No existing default is flipped — the webhook stays opt-in and webhook.enable stays false.

Broken today:

  • The controller cannot reach its own sidecar. The manager container does not set ADC_SERVER_URL, so the controller falls back to http://127.0.0.1:3000 while the ADC sidecar only listens on unix:/sockets/adc.sock. The Helm chart sets it.
  • The metrics Service points at a port nothing listens on. It exposes 8443/https, while the controller serves plain HTTP metrics on 8080 (metrics_addr: ":8080", secure_metrics: false), and the manager declares no container port at all. The ServiceMonitor scrapes the same missing port.
  • The webhook Service selects a label that does not exist. Its selector requires app.kubernetes.io/name: apisix-ingress-controller, but the pod template only carries control-plane: controller-manager, so the Service matches no endpoints. The webhook NetworkPolicy has the same mismatch.
  • The webhook patch is wrong. manager_webhook_patch.yaml passes --webhook-cert-path, a flag the controller does not have, and mounts the serving certificate at /tmp/k8s-webhook-server/serving-certs instead of /certs, which is what webhook.tls_cert_dir points at. The cert-manager CA injection the patch depends on was never wired up either, so uncommenting the [WEBHOOK] sections did not yield a working webhook.
  • certificate-metrics.yaml issues a certificate the controller cannot load — metrics TLS has no cert-directory option, so the Certificate is dead weight. Removed.

The [WEBHOOK] / [CERTMANAGER] sections stay commented out exactly as before; they now carry a correct patch and the cert-manager replacements, so uncommenting them (plus setting webhook.enable: true) produces a working webhook.

Layout and samples:

  • Moved the controller ConfigMap source from config/samples/config.yaml to config/manager/config.yaml. It is part of the deployment, not a sample. The file content is unchanged.
  • config/samples now covers all twelve CRDs (added BackendTrafficPolicy, L4RoutePolicy, PluginConfig) and the six # TODO(user): Add fields here stubs are filled in. Three samples were invalid against their own CRD schema: GatewayProxy was missing the required provider, and ApisixGlobalRule / ApisixPluginConfig were missing the required plugins.
  • Added the apiVersion/kind header to every kustomization.yaml, dropped leftover kubebuilder scaffolding comments, and made app.kubernetes.io/name consistently apisix-ingress-controller.

Verification

  • kustomize build config/default, config/crd and config/samples all succeed, and make build-installer regenerates dist/install.yaml.
  • Diffed the rendered config/default output against master. Apart from the app.kubernetes.io/name rename, the only changes are the four fixes above: the metrics Service now targets the named metrics port on 8080, the manager declares that port, ADC_SERVER_URL is set, and the pod carries the label the webhook Service and NetworkPolicy select on. The resource set is otherwise identical — no webhook or cert-manager objects are added.
  • Separately verified that uncommenting the [WEBHOOK] sections renders a ValidatingWebhookConfiguration with the cert-manager.io/inject-ca-from annotation and a Certificate whose dnsNames resolve to the prefixed webhook Service.
  • Validated every sample in config/samples against the CRD OpenAPI schema and its CEL x-kubernetes-validations rules (via a throwaway test built on the existing crdSchemaValidator harness in api/v2); all twelve pass.

CI

.github/workflows/kustomize-checker.yml guards config/ from here on. It runs when config/**, api/** or the Makefile changes and:

  1. runs make build-installer and builds config/crd and config/samples;
  2. checks config/crd/bases and config/webhook/manifests.yaml match what controller-gen produces (config/rbac/role.yaml is excluded — it has a pre-existing rule ordering difference against the pinned controller-gen);
  3. brings up a kind cluster and server-side applies the rendered installer, which validates every object against the API server, CRD schemas included;
  4. dry-run applies the samples against those CRDs, so a sample missing a required field or violating a CEL rule fails the job.

Every step above was run locally against a kind cluster before pushing: the installer applies cleanly and all twelve samples pass the server-side dry run.

`kustomize build config/default` produces a deployment that cannot talk to its
own sidecar and a metrics Service that points at nothing:

- The manager container does not set `ADC_SERVER_URL`, so the controller falls
  back to `http://127.0.0.1:3000` while the ADC sidecar only listens on
  `unix:/sockets/adc.sock`. The Helm chart sets it.
- `metrics_service.yaml` exposes 8443/https, but the controller serves plain
  HTTP metrics on 8080 (`metrics_addr: ":8080"`, `secure_metrics: false`), and
  the manager declares no container port at all. The ServiceMonitor scrapes the
  same missing port.
- The webhook Service selects `app.kubernetes.io/name: apisix-ingress-controller`,
  a label the pod template never carried, so it matches no endpoints. The webhook
  NetworkPolicy has the same mismatch.
- `manager_webhook_patch.yaml` passes `--webhook-cert-path`, a flag the
  controller does not have, and mounts the serving certificate at
  `/tmp/k8s-webhook-server/serving-certs` instead of the `/certs` directory
  `webhook.tls_cert_dir` points at. The cert-manager CA injection that the patch
  depends on was never wired up either.
- `certificate-metrics.yaml` issues a certificate the controller has no way to
  load: metrics TLS has no cert directory option.

The webhook stays opt-in and `webhook.enable` stays false, as before; the
[WEBHOOK] sections in config/default now describe how to turn it on and, unlike
before, produce a working webhook when uncommented.

Move the controller ConfigMap source from config/samples to config/manager,
since it is part of the deployment rather than a sample, and let config/samples
hold sample custom resources only. Complete the samples for all twelve CRDs and
fill in the empty specs: the GatewayProxy sample was missing the required
`provider` field, and the ApisixGlobalRule and ApisixPluginConfig samples were
missing the required `plugins` field, so applying them was rejected outright.

Add the apiVersion/kind header to every kustomization.yaml, drop the leftover
kubebuilder scaffolding comments and use `apisix-ingress-controller`
consistently for `app.kubernetes.io/name`.
@AlinsRan
AlinsRan force-pushed the chore/refresh-kustomize-manifests branch from eeb2b13 to 9a8a715 Compare August 5, 2026 05:32
@AlinsRan AlinsRan changed the title fix: make the kustomize manifests deployable and align them with the Helm chart fix: make the kustomize manifests deployable and consistent Aug 5, 2026
The kustomize manifests under config/ had no coverage: a broken patch, a
Service pointing at a port nothing listens on and samples that the API server
rejects outright all survived because nothing ever rendered or applied them.

Build every kustomization, then bring up a kind cluster and server-side apply
the rendered installer, which validates every object against the API server
including the CRD schemas. The samples are then dry-run applied against those
CRDs, so a sample missing a required field or violating a CEL rule fails the
job.

Also check that config/crd/bases and config/webhook/manifests.yaml match what
controller-gen produces. config/rbac/role.yaml is left out of that check: it
carries a pre-existing rule ordering difference against the pinned
controller-gen version.
The pipeline swallowed both a failed `kubectl get crd` and an empty grep
result, so the step passed without waiting for anything.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant