What is your environment, configuration, and command?
Macbook pro M4.
ProductName: macOS
ProductVersion: 15.1
BuildVersion: 24B2083
this is what i'm getting on
idpbuilder create --package https://github.com/cnoe-io/stacks//ref-implementation -l debug
What did you do and What did you see instead?
do the command..
and the error for the failed pods:
Failed to pull image "registry.k8s.io/coredns/coredns:v1.11.1": failed to pull and unpack image "registry.k8s.io/coredns/coredns:v1.11.1": failed to resolve reference "registry.k8s.io/coredns/coredns:v1.11.1": failed to do request: Head "https://registry.k8s.io/v2/coredns/coredns/manifests/v1.11.1": tls: failed to verify certificate: x509: certificate signed by unknown authority
when going into the docker container this is confirmed by
crictl pull registry.k8s.io/coredns/coredns:v1.11.1
pulling a nginx container works without problems..
edit:
I understood it's an issue with Zscaler proxy.
I added the following to the kind container in /etc/containerd/config.toml after [plugins."io.containerd.grpc.v1.cri".registry.configs]:
[plugins."io.containerd.grpc.v1.cri".registry.configs."k8s.io"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."k8s.io".tls]
insecure_skip_verify = true
[plugins."io.containerd.grpc.v1.cri".registry.configs."quay.io"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."quay.io".tls]
insecure_skip_verify = true
after
systemctl restart containerd
I am able to pull images.
I do understand it's not an automated solution.
Now I'm getting an error from keycloak config pod:
+ curl -sS -LO https://dl.k8s.io/release/v1.28.3//bin/linux/amd64/kubectl │
│ curl: (60) SSL certificate problem: unable to get local issuer certificate │
│ More details here: https://curl.se/docs/sslcerts.html │
│ │
│ curl failed to verify the legitimacy of the server and therefore could not │
│ establish a secure connection to it. To learn more about this situation and │
│ how to fix it, please visit the web page mentioned above. │
│ stream closed EOF for keycloak/config-5rzqm (kubectl)
Thinking about how to debug this
Additional Information. Logs.
idpbuilder create --package https://github.com/cnoe-io/stacks//ref-implementation -l debug
Jun 1 01:11:09 INFO Creating kind cluster logger=setup
########################### Our kind config ############################
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: "kindest/node:v1.31.4"
labels:
ingress-ready: "true"
extraPortMappings:
- containerPort: 443
hostPort: 8443
protocol: TCP
- containerPort: 32222
hostPort: 32222
protocol: TCP
containerdConfigPatches:
- |-
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."gitea.cnoe.localtest.me:8443"]
endpoint = ["https://gitea.cnoe.localtest.me"]
[plugins."io.containerd.grpc.v1.cri".registry.configs."gitea.cnoe.localtest.me".tls]
insecure_skip_verify = true
######################### config end ############################
Jun 1 01:11:09 INFO Creating kind cluster logger=setup cluster=localdev
Jun 1 01:11:09 DEBUG+3 Creating cluster "localdev" ...
logger=setup
Jun 1 01:11:09 DEBUG+3 • Ensuring node image (kindest/node:v1.31.4) 🖼 ...
logger=setup
Jun 1 01:11:09 DEBUG+2 Pulling image: kindest/node:v1.31.4 ... logger=setup
Jun 1 01:11:39 DEBUG+3 ✓ Ensuring node image (kindest/node:v1.31.4) 🖼
logger=setup
Jun 1 01:11:39 DEBUG+3 • Preparing nodes 📦 ...
logger=setup
Jun 1 01:11:40 DEBUG+3 ✓ Preparing nodes 📦
logger=setup
Jun 1 01:11:40 DEBUG+3 • Writing configuration 📜 ...
logger=setup
Jun 1 01:11:40 DEBUG+1 Using the following kubeadm config for node localdev-control-plane:
apiServer:
certSANs:
- localhost
- 127.0.0.1
extraArgs:
runtime-config: ""
apiVersion: kubeadm.k8s.io/v1beta3
clusterName: localdev
controlPlaneEndpoint: localdev-control-plane:6443
controllerManager:
extraArgs:
enable-hostpath-provisioner: "true"
kind: ClusterConfiguration
kubernetesVersion: v1.31.4
networking:
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/16
scheduler:
extraArgs: null
---
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- token: abcdef.0123456789abcdef
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 172.18.0.2
bindPort: 6443
nodeRegistration:
criSocket: unix:///run/containerd/containerd.sock
kubeletExtraArgs:
node-ip: 172.18.0.2
node-labels: ingress-ready=true
provider-id: kind://docker/localdev/localdev-control-plane
skipPhases:
- preflight
---
apiVersion: kubeadm.k8s.io/v1beta3
controlPlane:
localAPIEndpoint:
advertiseAddress: 172.18.0.2
bindPort: 6443
discovery:
bootstrapToken:
apiServerEndpoint: localdev-control-plane:6443
token: abcdef.0123456789abcdef
unsafeSkipCAVerification: true
kind: JoinConfiguration
nodeRegistration:
criSocket: unix:///run/containerd/containerd.sock
kubeletExtraArgs:
node-ip: 172.18.0.2
node-labels: ingress-ready=true
provider-id: kind://docker/localdev/localdev-control-plane
skipPhases:
- preflight
---
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
cgroupRoot: /kubelet
evictionHard:
imagefs.available: 0%
nodefs.available: 0%
nodefs.inodesFree: 0%
failSwapOn: false
imageGCHighThresholdPercent: 100
kind: KubeletConfiguration
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
conntrack:
maxPerCore: 0
iptables:
minSyncPeriod: 1s
kind: KubeProxyConfiguration
mode: iptables
logger=setup
Jun 1 01:11:40 DEBUG+3 ✓ Writing configuration 📜
logger=setup
Jun 1 01:11:40 DEBUG+3 • Starting control-plane 🕹️ ...
logger=setup
Jun 1 01:11:48 DEBUG I0531 22:11:41.080365 185 initconfiguration.go:261] loading configuration from "/kind/kubeadm.conf"
W0531 22:11:41.080602 185 common.go:101] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta3" (kind: "ClusterConfiguration"). Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
W0531 22:11:41.080812 185 common.go:101] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta3" (kind: "InitConfiguration"). Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
W0531 22:11:41.080952 185 common.go:101] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta3" (kind: "JoinConfiguration"). Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
W0531 22:11:41.081050 185 initconfiguration.go:361] [config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta3, Kind=JoinConfiguration
[init] Using Kubernetes version: v1.31.4
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I0531 22:11:41.081494 185 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I0531 22:11:41.110935 185 certs.go:473] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localdev-control-plane localhost] and IPs [10.96.0.1 172.18.0.2 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
I0531 22:11:41.203422 185 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I0531 22:11:41.250654 185 certs.go:473] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I0531 22:11:41.344371 185 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I0531 22:11:41.446645 185 certs.go:473] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localdev-control-plane localhost] and IPs [172.18.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localdev-control-plane localhost] and IPs [172.18.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I0531 22:11:41.782041 185 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
I0531 22:11:41.874440 185 kubeconfig.go:111] creating kubeconfig file for admin.conf
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
I0531 22:11:41.925715 185 kubeconfig.go:111] creating kubeconfig file for super-admin.conf
[kubeconfig] Writing "super-admin.conf" kubeconfig file
I0531 22:11:41.981752 185 kubeconfig.go:111] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I0531 22:11:42.012332 185 kubeconfig.go:111] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0531 22:11:42.100992 185 kubeconfig.go:111] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0531 22:11:42.123517 185 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I0531 22:11:42.123529 185 manifests.go:103] [control-plane] getting StaticPodSpecs
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I0531 22:11:42.123636 185 certs.go:473] validating certificate period for CA certificate
I0531 22:11:42.123670 185 manifests.go:129] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I0531 22:11:42.123675 185 manifests.go:129] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I0531 22:11:42.123676 185 manifests.go:129] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I0531 22:11:42.123678 185 manifests.go:129] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I0531 22:11:42.123678 185 manifests.go:129] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
I0531 22:11:42.124003 185 manifests.go:158] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
I0531 22:11:42.124011 185 manifests.go:103] [control-plane] getting StaticPodSpecs
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I0531 22:11:42.124076 185 manifests.go:129] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I0531 22:11:42.124082 185 manifests.go:129] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I0531 22:11:42.124084 185 manifests.go:129] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I0531 22:11:42.124085 185 manifests.go:129] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I0531 22:11:42.124086 185 manifests.go:129] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I0531 22:11:42.124087 185 manifests.go:129] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I0531 22:11:42.124088 185 manifests.go:129] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
I0531 22:11:42.124338 185 manifests.go:158] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
I0531 22:11:42.124345 185 manifests.go:103] [control-plane] getting StaticPodSpecs
[control-plane] Creating static Pod manifest for "kube-scheduler"
I0531 22:11:42.124393 185 manifests.go:129] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I0531 22:11:42.124549 185 manifests.go:158] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
I0531 22:11:42.124555 185 kubelet.go:68] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
I0531 22:11:42.178215 185 loader.go:395] Config loaded from file: /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 1.500694209s
[api-check] Waiting for a healthy API server. This can take up to 4m0s
I0531 22:11:43.679778 185 round_trippers.go:553] GET https://localdev-control-plane:6443/healthz?timeout=10s in 0 milliseconds
I0531 22:11:44.180218 185 round_trippers.go:553] GET https://localdev-control-plane:6443/healthz?timeout=10s in 0 milliseconds
I0531 22:11:46.001050 185 round_trippers.go:553] GET https://localdev-control-plane:6443/healthz?timeout=10s 403 Forbidden in 1321 milliseconds
I0531 22:11:46.001615 185 round_trippers.go:553] GET https://localdev-control-plane:6443/healthz?timeout=10s 403 Forbidden in 0 milliseconds
I0531 22:11:46.180694 185 round_trippers.go:553] GET https://localdev-control-plane:6443/healthz?timeout=10s 403 Forbidden in 0 milliseconds
I0531 22:11:46.682497 185 round_trippers.go:553] GET https://localdev-control-plane:6443/healthz?timeout=10s 403 Forbidden in 2 milliseconds
[api-check] The API server is healthy after 3.503290294s
I0531 22:11:47.182750 185 round_trippers.go:553] GET https://localdev-control-plane:6443/healthz?timeout=10s 200 OK in 0 milliseconds
I0531 22:11:47.183017 185 loader.go:395] Config loaded from file: /etc/kubernetes/admin.conf
I0531 22:11:47.183302 185 loader.go:395] Config loaded from file: /etc/kubernetes/super-admin.conf
I0531 22:11:47.183475 185 kubeconfig.go:665] ensuring that the ClusterRoleBinding for the kubeadm:cluster-admins Group exists
I0531 22:11:47.183945 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 403 Forbidden in 0 milliseconds
I0531 22:11:47.184034 185 kubeconfig.go:738] creating the ClusterRoleBinding for the kubeadm:cluster-admins Group by using super-admin.conf
I0531 22:11:47.187436 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 3 milliseconds
I0531 22:11:47.187567 185 uploadconfig.go:112] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I0531 22:11:47.188833 185 round_trippers.go:553] POST https://localdev-control-plane:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s 201 Created in 0 milliseconds
I0531 22:11:47.190274 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles?timeout=10s 201 Created in 1 milliseconds
I0531 22:11:47.191256 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings?timeout=10s 201 Created in 0 milliseconds
I0531 22:11:47.191304 185 uploadconfig.go:126] [upload-config] Uploading the kubelet component config to a ConfigMap
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
I0531 22:11:47.192276 185 round_trippers.go:553] POST https://localdev-control-plane:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s 201 Created in 0 milliseconds
I0531 22:11:47.193040 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles?timeout=10s 201 Created in 0 milliseconds
I0531 22:11:47.193680 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings?timeout=10s 201 Created in 0 milliseconds
I0531 22:11:47.193723 185 uploadconfig.go:131] [upload-config] Preserving the CRISocket information for the control-plane node
I0531 22:11:47.193732 185 patchnode.go:31] [patchnode] Uploading the CRI Socket information "unix:///run/containerd/containerd.sock" to the Node API object "localdev-control-plane" as an annotation
I0531 22:11:47.194338 185 round_trippers.go:553] GET https://localdev-control-plane:6443/api/v1/nodes/localdev-control-plane?timeout=10s 200 OK in 0 milliseconds
I0531 22:11:47.196348 185 round_trippers.go:553] PATCH https://localdev-control-plane:6443/api/v1/nodes/localdev-control-plane?timeout=10s 200 OK in 1 milliseconds
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node localdev-control-plane as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node localdev-control-plane as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
I0531 22:11:47.197057 185 round_trippers.go:553] GET https://localdev-control-plane:6443/api/v1/nodes/localdev-control-plane?timeout=10s 200 OK in 0 milliseconds
I0531 22:11:47.198985 185 round_trippers.go:553] PATCH https://localdev-control-plane:6443/api/v1/nodes/localdev-control-plane?timeout=10s 200 OK in 1 milliseconds
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
I0531 22:11:47.199980 185 round_trippers.go:553] GET https://localdev-control-plane:6443/api/v1/namespaces/kube-system/secrets/bootstrap-token-abcdef?timeout=10s 404 Not Found in 0 milliseconds
I0531 22:11:47.200834 185 round_trippers.go:553] POST https://localdev-control-plane:6443/api/v1/namespaces/kube-system/secrets?timeout=10s 201 Created in 0 milliseconds
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
I0531 22:11:47.201923 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?timeout=10s 201 Created in 0 milliseconds
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
I0531 22:11:47.202670 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 0 milliseconds
I0531 22:11:47.203734 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 1 milliseconds
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
I0531 22:11:47.204485 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 0 milliseconds
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
I0531 22:11:47.205245 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 0 milliseconds
I0531 22:11:47.205278 185 clusterinfo.go:47] [bootstrap-token] loading admin kubeconfig
I0531 22:11:47.205443 185 loader.go:395] Config loaded from file: /etc/kubernetes/admin.conf
I0531 22:11:47.205451 185 clusterinfo.go:58] [bootstrap-token] copying the cluster from admin.conf to the bootstrap kubeconfig
I0531 22:11:47.205543 185 clusterinfo.go:70] [bootstrap-token] creating/updating ConfigMap in kube-public namespace
I0531 22:11:47.206710 185 round_trippers.go:553] POST https://localdev-control-plane:6443/api/v1/namespaces/kube-public/configmaps?timeout=10s 201 Created in 1 milliseconds
I0531 22:11:47.206762 185 clusterinfo.go:84] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I0531 22:11:47.388931 185 request.go:632] Waited for 182.10925ms due to client-side throttling, not priority and fairness, request: POST:https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/roles?timeout=10s
I0531 22:11:47.391088 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/roles?timeout=10s 201 Created in 2 milliseconds
I0531 22:11:47.587165 185 request.go:632] Waited for 195.848459ms due to client-side throttling, not priority and fairness, request: POST:https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/rolebindings?timeout=10s
I0531 22:11:47.594956 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/rolebindings?timeout=10s 201 Created in 7 milliseconds
I0531 22:11:47.595377 185 kubeletfinalize.go:123] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0531 22:11:47.595958 185 loader.go:395] Config loaded from file: /etc/kubernetes/kubelet.conf
I0531 22:11:47.596504 185 kubeletfinalize.go:177] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
I0531 22:11:47.700444 185 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0531 22:11:47.700457 185 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0531 22:11:47.702510 185 round_trippers.go:553] GET https://localdev-control-plane:6443/apis/apps/v1/namespaces/kube-system/deployments?labelSelector=k8s-app%3Dkube-dns 200 OK in 1 milliseconds
I0531 22:11:47.704044 185 round_trippers.go:553] GET https://localdev-control-plane:6443/api/v1/namespaces/kube-system/configmaps/coredns?timeout=10s 404 Not Found in 0 milliseconds
I0531 22:11:47.705415 185 round_trippers.go:553] POST https://localdev-control-plane:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s 201 Created in 1 milliseconds
I0531 22:11:47.787955 185 request.go:632] Waited for 82.3865ms due to client-side throttling, not priority and fairness, request: POST:https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?timeout=10s
I0531 22:11:47.789305 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterroles?timeout=10s 201 Created in 1 milliseconds
I0531 22:11:47.986927 185 request.go:632] Waited for 197.342709ms due to client-side throttling, not priority and fairness, request: POST:https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s
I0531 22:11:47.992030 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 4 milliseconds
I0531 22:11:47.994258 185 round_trippers.go:553] POST https://localdev-control-plane:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s 201 Created in 1 milliseconds
I0531 22:11:47.998790 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/apps/v1/namespaces/kube-system/deployments?timeout=10s 201 Created in 3 milliseconds
I0531 22:11:48.002612 185 round_trippers.go:553] POST https://localdev-control-plane:6443/api/v1/namespaces/kube-system/services?timeout=10s 201 Created in 3 milliseconds
[addons] Applied essential addon: CoreDNS
I0531 22:11:48.004585 185 round_trippers.go:553] POST https://localdev-control-plane:6443/api/v1/namespaces/kube-system/configmaps?timeout=10s 201 Created in 1 milliseconds
I0531 22:11:48.007594 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/apps/v1/namespaces/kube-system/daemonsets?timeout=10s 201 Created in 2 milliseconds
I0531 22:11:48.190139 185 request.go:632] Waited for 182.284041ms due to client-side throttling, not priority and fairness, request: POST:https://localdev-control-plane:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s
I0531 22:11:48.196950 185 round_trippers.go:553] POST https://localdev-control-plane:6443/api/v1/namespaces/kube-system/serviceaccounts?timeout=10s 201 Created in 6 milliseconds
I0531 22:11:48.200321 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?timeout=10s 201 Created in 2 milliseconds
I0531 22:11:48.385141 185 request.go:632] Waited for 184.592667ms due to client-side throttling, not priority and fairness, request: POST:https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles?timeout=10s
I0531 22:11:48.389441 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/roles?timeout=10s 201 Created in 3 milliseconds
I0531 22:11:48.587797 185 request.go:632] Waited for 198.114708ms due to client-side throttling, not priority and fairness, request: POST:https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings?timeout=10s
I0531 22:11:48.595437 185 round_trippers.go:553] POST https://localdev-control-plane:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings?timeout=10s 201 Created in 7 milliseconds
[addons] Applied essential addon: kube-proxy
I0531 22:11:48.596193 185 loader.go:395] Config loaded from file: /etc/kubernetes/admin.conf
I0531 22:11:48.596499 185 loader.go:395] Config loaded from file: /etc/kubernetes/admin.conf
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
kubeadm join localdev-control-plane:6443 --token <value withheld> \
--discovery-token-ca-cert-hash sha256:318e7a990932caa004d40796da8046aaf5e3418c6e247c19208a63a0e75000fd \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join localdev-control-plane:6443 --token <value withheld> \
--discovery-token-ca-cert-hash sha256:318e7a990932caa004d40796da8046aaf5e3418c6e247c19208a63a0e75000fd logger=setup
Jun 1 01:11:48 DEBUG+3 ✓ Starting control-plane 🕹️
logger=setup
Jun 1 01:11:48 DEBUG+3 • Installing CNI 🔌 ...
logger=setup
Jun 1 01:11:49 DEBUG+3 ✓ Installing CNI 🔌
logger=setup
Jun 1 01:11:49 DEBUG+3 • Installing StorageClass 💾 ...
logger=setup
Jun 1 01:11:49 DEBUG+3 ✓ Installing StorageClass 💾
logger=setup
Jun 1 01:11:49 INFO Done creating cluster logger=setup cluster=localdev
Jun 1 01:11:49 DEBUG+3 Getting Kube config logger=setup
Jun 1 01:11:49 DEBUG+3 Getting Kube client logger=setup
Jun 1 01:11:49 INFO Adding CRDs to the cluster logger=setup
Jun 1 01:11:49 DEBUG+3 crd not yet established, waiting. crd name=custompackages.idpbuilder.cnoe.io
Jun 1 01:11:49 DEBUG+3 crd not yet established, waiting. crd name=custompackages.idpbuilder.cnoe.io
Jun 1 01:11:49 DEBUG+3 crd not yet established, waiting. crd name=custompackages.idpbuilder.cnoe.io
Jun 1 01:11:49 DEBUG+3 crd not yet established, waiting. crd name=gitrepositories.idpbuilder.cnoe.io
Jun 1 01:11:49 DEBUG+3 crd not yet established, waiting. crd name=gitrepositories.idpbuilder.cnoe.io
Jun 1 01:11:50 DEBUG+3 Creating controller manager logger=setup
Jun 1 01:11:50 DEBUG+3 Created temp directory for cloning repositories logger=setup dir=/var/folders/22/v1_ljx012_l6_rhgk86srlvw0000gn/T/idpbuilder-localdev-797909713
Jun 1 01:11:50 INFO Setting up CoreDNS logger=setup
Jun 1 01:11:50 INFO Setting up TLS certificate logger=setup
Jun 1 01:11:50 DEBUG+3 Creating/getting certificate logger=setup host=cnoe.localtest.me sans=[cnoe.localtest.me *.cnoe.localtest.me]
Jun 1 01:11:50 DEBUG+3 Creating secret for certificate logger=setup host=cnoe.localtest.me
Jun 1 01:11:50 DEBUG+3 Creating secret for ArgoCD server logger=setup host=cnoe.localtest.me
Jun 1 01:11:50 DEBUG+3 Checking for incompatible options from a previous run logger=setup
Jun 1 01:11:50 DEBUG+3 Running controllers logger=setup
Jun 1 01:11:50 DEBUG+3 starting manager
Jun 1 01:11:50 INFO Creating localbuild resource logger=setup
Jun 1 01:11:50 INFO Starting EventSource controller=custompackage controllerGroup=idpbuilder.cnoe.io controllerKind=CustomPackage source=kind source: *v1alpha1.CustomPackage
Jun 1 01:11:50 INFO Starting EventSource controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild source=kind source: *v1alpha1.Localbuild
Jun 1 01:11:50 INFO Starting EventSource controller=gitrepository controllerGroup=idpbuilder.cnoe.io controllerKind=GitRepository source=kind source: *v1alpha1.GitRepository
Jun 1 01:11:50 INFO Starting Controller controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild
Jun 1 01:11:50 INFO Starting Controller controller=gitrepository controllerGroup=idpbuilder.cnoe.io controllerKind=GitRepository
Jun 1 01:11:50 INFO Starting Controller controller=custompackage controllerGroup=idpbuilder.cnoe.io controllerKind=CustomPackage
Jun 1 01:11:50 INFO Starting workers controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild worker count=1
Jun 1 01:11:50 INFO Starting workers controller=custompackage controllerGroup=idpbuilder.cnoe.io controllerKind=CustomPackage worker count=1
Jun 1 01:11:50 INFO Starting workers controller=gitrepository controllerGroup=idpbuilder.cnoe.io controllerKind=GitRepository worker count=1
Jun 1 01:11:52 DEBUG+3 Reconciling controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=82bd0d19-68f4-4689-bf97-9fa7de7ce420 resource=/localdev
Jun 1 01:11:52 DEBUG+3 Create or update namespace controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=82bd0d19-68f4-4689-bf97-9fa7de7ce420 resource=&Namespace{ObjectMeta:{idpbuilder-localdev 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:NamespaceSpec{Finalizers:[],},Status:NamespaceStatus{Phase:,Conditions:[]NamespaceCondition{},},}
Jun 1 01:11:52 DEBUG+3 installing core packages controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=82bd0d19-68f4-4689-bf97-9fa7de7ce420
Jun 1 01:11:53 DEBUG+3 failed installing controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=82bd0d19-68f4-4689-bf97-9fa7de7ce420 name=gitea error=Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": EOF
Jun 1 01:11:53 DEBUG+3 failed installing core package. likely not fatal. will try again controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=82bd0d19-68f4-4689-bf97-9fa7de7ce420 error=failed installing gitea: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": EOF
Jun 1 01:11:53 INFO Checking if we should shutdown controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=82bd0d19-68f4-4689-bf97-9fa7de7ce420
Jun 1 01:11:53 ERROR failed to reconcile the Nginx resources controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=82bd0d19-68f4-4689-bf97-9fa7de7ce420 err=Timeout: failed waiting for *v1.Deployment Informer to sync
Jun 1 01:11:53 DEBUG+3 failed installing controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=82bd0d19-68f4-4689-bf97-9fa7de7ce420 name=argocd error=Post "https://127.0.0.1:57292/apis/apiextensions.k8s.io/v1/customresourcedefinitions": context canceled
Jun 1 01:11:53 DEBUG+3 failed installing controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=82bd0d19-68f4-4689-bf97-9fa7de7ce420 name=nginx error=Timeout: failed waiting for *v1.Deployment Informer to sync
Jun 1 01:11:53 DEBUG+3 Reconciling controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=d8ebaf3b-87eb-40a8-9faa-d6674a441ad0 resource=/localdev
Jun 1 01:11:53 DEBUG+3 Create or update namespace controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=d8ebaf3b-87eb-40a8-9faa-d6674a441ad0 resource=&Namespace{ObjectMeta:{idpbuilder-localdev 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},Spec:NamespaceSpec{Finalizers:[],},Status:NamespaceStatus{Phase:,Conditions:[]NamespaceCondition{},},}
Jun 1 01:11:53 DEBUG+3 installing core packages controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=d8ebaf3b-87eb-40a8-9faa-d6674a441ad0
Jun 1 01:11:53 INFO Waiting for Deployment ingress-nginx-controller to become ready controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=d8ebaf3b-87eb-40a8-9faa-d6674a441ad0
Jun 1 01:11:53 DEBUG+3 failed installing controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=d8ebaf3b-87eb-40a8-9faa-d6674a441ad0 name=gitea error=Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": EOF
Jun 1 01:11:53 DEBUG+3 failed installing core package. likely not fatal. will try again controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=d8ebaf3b-87eb-40a8-9faa-d6674a441ad0 error=failed installing gitea: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook: Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": EOF
Jun 1 01:11:53 INFO Checking if we should shutdown controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=d8ebaf3b-87eb-40a8-9faa-d6674a441ad0
Jun 1 01:11:53 DEBUG+3 failed installing controller=localbuild controllerGroup=idpbuilder.cnoe.io controllerKind=Localbuild name=localdev name=localdev reconcileID=d8ebaf3b-87eb-40a8-9faa-d6674a441ad0 name=argocd error=client rate limiter Wait returned an error: context canceled
What is your environment, configuration, and command?
Macbook pro M4.
ProductName: macOS
ProductVersion: 15.1
BuildVersion: 24B2083
this is what i'm getting on
What did you do and What did you see instead?
do the command..
and the error for the failed pods:
when going into the docker container this is confirmed by
pulling a nginx container works without problems..
edit:
I understood it's an issue with Zscaler proxy.
I added the following to the kind container in /etc/containerd/config.toml after [plugins."io.containerd.grpc.v1.cri".registry.configs]:
after
I am able to pull images.
I do understand it's not an automated solution.
Now I'm getting an error from keycloak config pod:
Thinking about how to debug this
Additional Information. Logs.