Control plane deadlock via Gatekeeper-Generated ValidatingAdmissionPolicies

[shikanime]

What steps did you take and what happened:
I am using k0s with Gatekeeper integration with Gatekeeper Library including (kinda specifically those):

• K8sPSPPrivilegedContainer
• K8sPSPHostProcess
• K8sPSPAllowPrivilegeEscalationContainer.

Source of my Gatekeeper configurations: https://github.com/shikanime/manifests/tree/6aaa5a792a96f68abb4ad9741af3af37b78242fa/configs/gatekeeper.

After a cluster restart, the control plane failed to bootstrap. The kube-apiserver became active but immediately began denying critical internal k0s and Kubernetes management requests. Specifically, it blocked the creation of EtcdMember objects and the update of Leases in kube-node-lease.

Because the VAP was configured to use a Gatekeeper Constraint as a paramKind (constraints.gatekeeper.sh/v1beta1), and those constraints were not yet fully initialized or reachable during the early boot phase, the API server "failed closed."

This created a circular dependency:

1. k0s needs to create an EtcdMember to start the cluster database.
2. The API server intercepts this request to check a VAP.
3. The VAP fails to find its parameter definition (Constraint).
4. The API server denies the EtcdMember creation.
5. The cluster database never starts, keeping the cluster in a permanent "Connection Refused" death loop.

What did you expect to happen:

Perhaps we should add something specific to the documentation for distribution-specific installations. I don't remember if RKE2 or K3s does anything similar to managing the cluster state within the cluster.

Anything else you would like to add:

The logs show the API server explicitly forbidding the k0s supervisor from managing the cluster:

E0422 22:11:26.971959 1804 leaderelection.go:488] "Failed to update lease" err="leases.coordination.k8s.io \"k0s-endpoint-reconciler\" is forbidden: ValidatingAdmissionPolicy 'gatekeeper-k8spspprivilegedcontainer' denied request: failed to configure policy: failed to find resource referenced by paramKind: 'constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPPrivilegedContainer'"
Error: failed to start cluster components: failed to create EtcdMember object for this controller: etcdmembers.etcd.k0sproject.io "nishir" is forbidden: ValidatingAdmissionPolicy 'gatekeeper-k8spspprivilegedcontainer' denied request...

Temporary mitigation:

sudo k0s kubectl delete validatingadmissionpolicy --all
sudo k0s kubectl delete validatingwebhookconfiguration -l gatekeeper.sh/system=yes
sudo k0s kubectl delete mutatingwebhookconfiguration -l gatekeeper.sh/system=yes

Environment:

• Gatekeeper version: 3.22.0
• Kubernetes version:
  • Client Version: v1.35.2
  • Kustomize Version: v5.7.1
  • Server Version: v1.35.2+k0s

[JaydipGabani]

@shikanime thanks for opening this issue. would you like to contribute to adding something in the docs for this? Only other solutions to address this I can think of is add a flag --default-to-fail-close-after which allows users to configure a time after restart for which the failure policy is set to Ignore and then flips to Fail after sometime. However, I am not sure if this is a good solution. Open to ideas/suggestions.

[bovy89]

We are also facing the same issue while upgrading Kubernetes from 1.32 → 1.33 with Open Policy Agent Gatekeeper version 3.22.1.

Error logs

encountered error syncing policies: failed to find resource referenced by paramKind: 'constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPCapabilities'
failed to find resource referenced by paramKind: 'constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPForbiddenSysctls'
failed to find resource referenced by paramKind: 'constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPHostNamespace'
failed to find resource referenced by paramKind: 'constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPHostNetworkingPorts'
failed to find resource referenced by paramKind: 'constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPPrivilegedContainer'
failed to find resource referenced by paramKind: 'constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPProcMount'
failed to find resource referenced by paramKind: 'constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPAllowPrivilegeEscalationContainer'. Rescheduling policy sync

Additional logs:

I0428 policy_source.go:240] refreshing policies
I0428 policy_source.go:435] informer started for constraints.gatekeeper.sh/v1beta1, Kind=K8sPSPProcMount
...
F0428 hooks.go:204] PostStartHook "start-service-ip-repair-controllers" failed: unable to perform initial IP and Port allocation check

As temporary mitigation we found that kubectl delete constrainttemplate <name> works as a workaround. Deleting only the ValidatingAdmissionPolicy is ineffective because it is immediately recreated. Removing the ConstraintTemplate stops the regeneration loop and restores cluster functionality. kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io gatekeeper-validating-webhook-configuration seems useless for us.

[bovy89]

I’m currently working around the issue by adding the following patch:

- op: add
  path: "/spec/template/spec/containers/0/args/-"
  value: --default-create-vap-for-templates=false
- op: add
  path: "/spec/template/spec/containers/0/args/-"
  value: --default-create-vap-binding-for-constraints=false

This seems to work in my setup
Do you think this is a viable/acceptable solution, or am I just masking the underlying problem?

[JaydipGabani]

It is a reasonable solution, but if you ever use VAP resource with paramKind that is a CRD you will face this issue regardless even outside of Gatekeeper. Better solution is to set failure policy on VAP and CT to ignore, so that you can still use VAP enforcement but not end up in failure state for situation like this.

[bovy89]

It is a reasonable solution, but if you ever use VAP resource with paramKind that is a CRD you will face this issue regardless even outside of Gatekeeper. Better solution is to set failure policy on VAP and CT to ignore, so that you can still use VAP enforcement but not end up in failure state for situation like this.

do you mean something like that:

patches:
  - target:
      kind: ConstraintTemplate
      name: k8spspallowprivilegeescalationcontainer
    patch: |-
      - op: add
        path: /spec/targets/0/code/0/source/failurePolicy
        value: "Ignore"

instead of those parameters:

- op: add
  path: "/spec/template/spec/containers/0/args/-"
  value: --default-create-vap-for-templates=false
- op: add
  path: "/spec/template/spec/containers/0/args/-"
  value: --default-create-vap-binding-for-constraints=false

[JaydipGabani]

something like this per constraint template that generates vap -

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabelsvap
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabelsVap
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          type: object
          properties:
            message:
              type: string
            labels:
              type: array
              items:
                type: object
                properties:
                  key:
                    type: string
                  allowedRegex:
                    type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      code:
        - engine: K8sNativeValidation
          source:
            generateVAP: true
            failurePolicy: Ignore ## <- this is the field to control failure policy
            validations:
            - expression: '[object, oldObject].exists(obj, obj != null && has(obj.metadata) && variables.params.labels.all(entry, has(obj.metadata.labels) && entry.key in obj.metadata.labels))'
              messageExpression: '"missing required label, requires all of: " + variables.params.labels.map(entry, entry.key).join(", ")'
            - expression: '[object, oldObject].exists(obj, obj != null && !variables.params.labels.exists(entry, has(obj.metadata.labels) && entry.key in obj.metadata.labels && !string(obj.metadata.labels[entry.key]).matches(string(entry.allowedRegex))))'
              message: "regex mismatch"

[charleswool]

I think the flag in #4568 is a necessary step but doesn't work for users running the stock Library templates with default settings. May be we can consider decoupling the default for generated VAPs from the in-process webhook. In TemplateToPolicyDefinitionWithWebhookConfig, we default the generated VAP's failurePolicy to Ignore when the template omits it,. The webhook remains the authoritative enforcer while the VAP becomes a performance optimization that simply skips when its params aren't resolvable, rather than bricking the cluster.

[JaydipGabani]

In TemplateToPolicyDefinitionWithWebhookConfig, we default the generated VAP's failurePolicy to Ignore when the template omits it

This will be a breaking change for the existing beta users. As well as upstream k8s uses Fail as failure policy when the field is not set, so we should follow the same as it may be expected behavior from policy authors. So changing this default should be the last resort.

@charleswool can you elaborate why the flag in #4568 doesn't work for users running the stock Library templates with default settings?

[charleswool]

In https://github.com/open-policy-agent/gatekeeper/pull/4568/changes#diff-4d485fc3144c980825c8c531fa38e3e788cec4047963df5de63156689573ff61R17 , it defaults --default-k8s-native-validation-failure-\to Fail, which is a knob but does not change:

• --default-create-vap-for-templates=true for new install of GK 3.22+
• users may only applies stock Library templates without knowing failurePolicy exisits
• generated VAPs are fail-closed against / since the new flag is default to false.

With above being said, this change only helps when user read the document and opt-in to ingore. However, if a naive user just follows the README to install Gatekeeper + Library defaults, the issue could still happen.

Ack this is going to be an breaking change though...Do you think we can exclude kube system namespaces from the generated VAPs' matchConstraints, as long as we document this clearly, it won't be considered as a violation of compliance. @JaydipGabani