#373 appears to break spegel chart functionality

[billimek]

Spegel version

0.1.1

Kubernetes distribution

K3S

Kubernetes version

v1.31.7

CNI

flannel

Describe the bug

#373 appears to render spegel chart not usable, at least in k3s:

After upgrading from v0.30.0 to 0.1.1 k3s and containerd no longer find images hosted in the removed public upstream registries.

See these details for the exact error messages and eventual remedy of reverting spegel chart to the earlier v0.30.0 version

[billimek]

Forgot to reference the spegel helm chart values used:

    spegel:
      containerdSock: /run/k3s/containerd/containerd.sock
      containerdRegistryConfigPath: /var/lib/rancher/k3s/agent/etc/containerd/certs.d
      containerdContentPath: /var/lib/rancher/k3s/agent/containerd/io.containerd.content.v1.content
    serviceMonitor:
      enabled: true
    grafanaDashboard:
      enabled: true
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        memory: 1Gi

[billimek]

Some more details,

running spegel helm chart v0.30.0 in k3s, examining /var/lib/rancher/k3s/agent/etc/containerd/certs.d:

[root@k3s-h:/var/lib/rancher/k3s/agent/etc/containerd/certs.d]# ll
total 44
drwxr-xr-x 12 root root 4096 Oct  8 12:32 _backup
drwxr-xr-x  2 root root 4096 Mar 30 20:31 cgr.dev
drwxr-xr-x  2 root root 4096 Mar 30 20:31 docker.io
drwxr-xr-x  2 root root 4096 Mar 30 20:31 gcr.io
drwxr-xr-x  2 root root 4096 Mar 30 20:31 ghcr.io
drwxr-xr-x  2 root root 4096 Mar 30 20:31 k8s.gcr.io
drwxr-xr-x  2 root root 4096 Mar 30 20:31 lscr.io
drwxr-xr-x  2 root root 4096 Mar 30 20:31 mcr.microsoft.com
drwxr-xr-x  2 root root 4096 Mar 30 20:31 public.ecr.aws
drwxr-xr-x  2 root root 4096 Mar 30 20:31 quay.io
drwxr-xr-x  2 root root 4096 Mar 30 20:31 registry.k8s.io

[root@k3s-h:/var/lib/rancher/k3s/agent/etc/containerd/certs.d]# cat quay.io/hosts.toml
server = 'https://quay.io'

[host.'http://10.2.0.41:30020']
capabilities = ['pull', 'resolve']

[host.'http://10.2.0.41:30021']
capabilities = ['pull', 'resolve']

... after upgrading spegel helm chart to 0.1.1 in k3s, examining the same host directory:

[root@k3s-h:/var/lib/rancher/k3s/agent/etc/containerd/certs.d]# ll
total 8
drwxr-xr-x 12 root root 4096 Oct  8 12:32 _backup
drwxr-xr-x  2 root root 4096 Mar 31 12:05 _default

[root@k3s-h:/var/lib/rancher/k3s/agent/etc/containerd/certs.d]# cat _default/hosts.toml
[host.'http://10.2.0.41:30020']
capabilities = ['pull', 'resolve']

[host.'http://10.2.0.41:30021']
capabilities = ['pull', 'resolve']

This would suggest that all registries should be mirrored as #373 implies, but containerd pulls begin to fail for known and valid images:

ubuntu@k3s-d:~$ sudo crictl --debug pull ghcr.io/spegel-org/spegel:v0.1.0
DEBU[0000] Asset dir /var/lib/rancher/k3s/data/817ebd4deb914cf102d29caf91a8b15c677f99fc7270d717991dc4d9eca7fa70
DEBU[0000] Running /var/lib/rancher/k3s/data/817ebd4deb914cf102d29caf91a8b15c677f99fc7270d717991dc4d9eca7fa70/bin/crictl [crictl --debug pull ghcr.io/spegel-org/spegel:v0.1.0]
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:ghcr.io/spegel-org/spegel:v0.1.0,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},Auth:nil,SandboxConfig:nil,}
E0331 12:07:04.239173  110044 log.go:32] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"ghcr.io/spegel-org/spegel:v0.1.0\": failed to resolve reference \"ghcr.io/spegel-org/spegel:v0.1.0\": ghcr.io/spegel-org/spegel:v0.1.0: not found" image="ghcr.io/spegel-org/spegel:v0.1.0"
FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/spegel-org/spegel:v0.1.0": failed to resolve reference "ghcr.io/spegel-org/spegel:v0.1.0": ghcr.io/spegel-org/spegel:v0.1.0: not found

after reverting to spegel helm chart v0.30.0,

ubuntu@k3s-d:~$ sudo crictl --debug pull ghcr.io/spegel-org/spegel:v0.1.0
DEBU[0000] Asset dir /var/lib/rancher/k3s/data/817ebd4deb914cf102d29caf91a8b15c677f99fc7270d717991dc4d9eca7fa70
DEBU[0000] Running /var/lib/rancher/k3s/data/817ebd4deb914cf102d29caf91a8b15c677f99fc7270d717991dc4d9eca7fa70/bin/crictl [crictl --debug pull ghcr.io/spegel-org/spegel:v0.1.0]
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:ghcr.io/spegel-org/spegel:v0.1.0,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},Auth:nil,SandboxConfig:nil,}
DEBU[0000] PullImageResponse: &PullImageResponse{ImageRef:sha256:cb4a8c3778ec55a584ac3e40c3acc1fc5499349fefb1bce786aab4c0fa26e49f,}
Image is up to date for sha256:cb4a8c3778ec55a584ac3e40c3acc1fc5499349fefb1bce786aab4c0fa26e49f

[phillebaba]

As far as I know k3s does some special things with their embedded Containerd implementation, which is why it is embedded in k3s. I dont see how k3s would have an issue with the default registry configuration as they have been using it longer than I have. I verified that this is not an issue in kind, I would have to setup a k3s cluster to recreate your issues.

Is there a reason that you are not using the embedded Spegel in k3s?

[phillebaba]

@billimek have you tried setting the registry list as it was before using Spegel 0.1.1 to see if that works?

[billimek]

Hi @phillebaba unfortunately I've seen migrated from k3s to talos and can't reproduce the issue there any longer.

[cook1emr]

Hi @phillebaba unfortunately I've seen migrated from k3s to talos and can't reproduce the issue there any longer.

I can reproduce the issue and will try @phillebaba solution and report back.

[cook1emr]

unfortunately applying the suggestion, still results in the same behaviour that @billimek is reporting

ubuntu@k3s-d:~$ sudo crictl --debug pull ghcr.io/spegel-org/spegel:v0.1.0
DEBU[0000] Asset dir /var/lib/rancher/k3s/data/817ebd4deb914cf102d29caf91a8b15c677f99fc7270d717991dc4d9eca7fa70
DEBU[0000] Running /var/lib/rancher/k3s/data/817ebd4deb914cf102d29caf91a8b15c677f99fc7270d717991dc4d9eca7fa70/bin/crictl [crictl --debug pull ghcr.io/spegel-org/spegel:v0.1.0]
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:ghcr.io/spegel-org/spegel:v0.1.0,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},Auth:nil,SandboxConfig:nil,}
E0331 12:07:04.239173 110044 log.go:32] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/spegel-org/spegel:v0.1.0": failed to resolve reference "ghcr.io/spegel-org/spegel:v0.1.0": ghcr.io/spegel-org/spegel:v0.1.0: not found" image="ghcr.io/spegel-org/spegel:v0.1.0"
FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "ghcr.io/spegel-org/spegel:v0.1

[cook1emr]

update:

explicitly setting the registries within the values appears to resolve the pull issues

mirroredRegistries:
        - https://cgr.dev
        - https://docker.io
        - https://ghcr.io
        - https://quay.io
        - https://mcr.microsoft.com
        - https://public.ecr.aws
        - https://gcr.io
        - https://registry.k8s.io
        - https://k8s.gcr.io
        - https://lscr.io