Issues with configuring mirror regisitries - response was http.StatusNotFound #5478
Comments
|
Just to get the complete config, can you maybe also share the contents of |
|
containerd.toml # k0s_managed=true
# This is a placeholder configuration for k0s managed containerd.
# If you wish to override the config, remove the first line and replace this file with your custom configuration.
# For reference see https://github.com/containerd/containerd/blob/main/docs/man/containerd-config.toml.5.md
version = 2
imports = [
"/run/k0s/containerd-cri.toml",
]run/k0s/containerd-cri.toml version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]
device_ownership_from_security_context = false
disable_apparmor = false
disable_cgroup = false
disable_hugetlb_controller = true
disable_proc_mount = false
disable_tcp_service = true
drain_exec_sync_io_timeout = "0s"
enable_cdi = false
enable_selinux = false
enable_tls_streaming = false
enable_unprivileged_icmp = false
enable_unprivileged_ports = false
ignore_deprecation_warnings = []
ignore_image_defined_volumes = false
image_pull_progress_timeout = "5m0s"
image_pull_with_sync_fs = false
max_concurrent_downloads = 3
max_container_log_line_size = 16384
netns_mounts_under_state_dir = false
restrict_oom_score_adj = false
sandbox_image = "registry.k8s.io/pause:3.9"
selinux_category_range = 1024
stats_collect_period = 10
stream_idle_timeout = "4h0m0s"
stream_server_address = "127.0.0.1"
stream_server_port = "0"
systemd_cgroup = false
tolerate_missing_hugetlb_controller = true
unset_seccomp_profile = ""
[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"
conf_template = ""
ip_pref = ""
max_conf_num = 1
setup_serially = false
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
disable_snapshot_annotations = true
discard_unpacked_layers = false
ignore_blockio_not_enabled_errors = false
ignore_rdt_not_enabled_errors = false
no_pivot = false
snapshotter = "overlayfs"
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
privileged_without_host_devices_all_devices_allowed = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = ""
sandbox_mode = ""
snapshotter = ""
[plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
privileged_without_host_devices_all_devices_allowed = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = "io.containerd.runc.v2"
sandbox_mode = "podsandbox"
snapshotter = ""
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
BinaryName = ""
CriuImagePath = ""
CriuPath = ""
CriuWorkPath = ""
IoGid = 0
IoUid = 0
NoNewKeyring = false
NoPivotRoot = false
Root = ""
ShimCgroup = ""
SystemdCgroup = false
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
base_runtime_spec = ""
cni_conf_dir = ""
cni_max_conf_num = 0
container_annotations = []
pod_annotations = []
privileged_without_host_devices = false
privileged_without_host_devices_all_devices_allowed = false
runtime_engine = ""
runtime_path = ""
runtime_root = ""
runtime_type = ""
sandbox_mode = ""
snapshotter = ""
[plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]
[plugins."io.containerd.grpc.v1.cri".image_decryption]
key_model = "node"
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/k0s/certs.d"
[plugins."io.containerd.grpc.v1.cri".registry.auths]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.headers]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
tls_cert_file = ""
tls_key_file = "" |
|
hmm, I think containerd gets somehow tripped over this: I see your config does set
|
|
So the CA Root is in the OS level trust record, does K0S not use the OS level ca root chain? I'm going to burn and rebuild just to make sure I don't have anything else going on but it doesn't make a whole lot of sense to me, it looks like it's pulling in just part of the config. |
Ah, didn't realize that is the case, my bad. Containerd should use the OS level CAs. Yu could of course test by adding the CA into the config directly, just to rule out things. One thing I'm not sure of is that you use You could also increase the containerd log level to see if that reveals anything how it handles the config. Edit the systems unit to include something like: |
twz123
added
question
|
Weird. If I understand the containerd logs correctly, containerd is trying to pull twice from registry.home.lab:
This was the first attempt. Apparently all went well, but the registry didn't find the tag.
This was the second attempt in which TLS verification failed. Why would it try two hosts? Maybe one for the exact match on registry.home.lab, and the other for the _default host? Even then, why would the first succeed, and the second one fail. As far as I can see, the two config snippets are identical. One remarkable thing is the URL that has been tried: Could you try to NB: I noticed that in the
You mean skip_verify, right? |
|
@jnummelin thanks for the command for debug, I'll get that going with my next test. The @twz123 yes
Yeah I noticed that also, not sure if this is a doc example short coming or a possible sign that
So I test this:
With this knowledge I tried to do my with Warning Failed 7m6s (x6 over 8m29s) kubelet Error: ImagePullBackOff
Normal Pulling 6m55s (x4 over 8m29s) kubelet Pulling image "registry.home.lab/nginxinc/nginx-unprivileged:1.25.1"
Warning Failed 6m55s (x4 over 8m29s) kubelet Failed to pull image "registry.home.lab/nginxinc/nginx-unprivileged:1.25.1": failed to pull and unpack image "registry.home.lab/nginxinc/nginx-unprivileged:1.25.1": failed to resolve reference "registry.home.lab/nginxinc/nginx-unprivileged:1.25.1": failed to do request: Head "https://registry.home.lab/v2/nginxinc/nginx-unprivileged/manifests/1.25.1": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match registry.home.labwith Normal Scheduled 4m28s default-scheduler Successfully assigned default/test2 to worker0.home.lab
Warning Failed 3m8s (x6 over 4m27s) kubelet Error: ImagePullBackOff
Normal Pulling 2m53s (x4 over 4m28s) kubelet Pulling image "registry.home.lab:5000/nginxinc/nginx-unprivileged:1.25.1"
Warning Failed 2m53s (x4 over 4m27s) kubelet Failed to pull image "registry.home.lab:5000/nginxinc/nginx-unprivileged:1.25.1": failed to pull and unpack image "registry.home.lab:5000/nginxinc/nginx-unprivileged:1.25.1": failed to resolve reference "registry.home.lab:5000/nginxinc/nginx-unprivileged:1.25.1": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentialsI think I did find one issue with the TLS cert for my registry. Appears my automation for handling this as snuck the port into the SAN, so i'll get that fixed. Next is to redeploy with the updated debugging. EDIT: to me it looks like it's using part of the |
|
Here's some more logs with the containerd=debug @twz123 Before you ask I did try curling my mirror registry for |
|
Given that you're now including the port, did you rename the respective config directory as well? IIUC it needs to be named https://github.com/containerd/containerd/blob/v1.7.25/remotes/docker/config/hosts.go#L291-L298 Did you remove
That's the URL without /v2/ (supposedly because
That's the right URL, but the special config didn't kick in (there's no corresponding "loading host directory" log entry), and hence it gets the unauthorized response. |
|
So I think I got this sorted out. I'll have a follow up with notes once i'm done testing. |
Nice to hear 🎉 , I hope I'm not celebrating too early... 😂
I think a good example in the docs would be fantastic to have, care to submit a PR once you have all the things sorted out? |
|
@jnummelin I'll try to get a PR submitted this week with some doc updates. Can I use this issue for that update or does the project like a specific issue to track commits and PRs? |
@killergoalie You can go ahead and use this issue 👍 |
Hi I'm running into a not so fun issue with trying to configure ContainerD to use a mirror registry (well several or the _default) configs.
Currently running k0s 1.30.5 in an airgapped/network restricted setup, using the tar ball works without an issue. With RHEL 8.10 kernel is
4.18.0-553.16.1.el8_10.x86_64.But trying to setup the registry mirror using either v1 or v2/v3 configs for ContainerD I just keep getting the same in the logs:
trying next host - response was http.StatusNotFoundIt's also ignoring the skip_secure for some reason.
Full log (trying to deploy from
registry.lab/nginxinc/nginx-unprivileged:v1.25.1):Full log (trying to deploy from
nginxinc/nginx-unprivileged:v1.25.1idea being to let the registry config pull from the mirror):Current config:
/etc/k0s/containderd.d/registry.tomlTree for
/etc/k0s/certs.d[root@worker0 certs.d]# tree /etc/k0s/certs.d/ /etc/k0s/certs.d/ ├── registry.home.lab │ └── hosts.toml ├── _default │ └── hosts.toml ├── docker.io │ └── hosts.toml ├── quay.io │ └── hosts.toml └── registry.k8s.io └── hosts.toml 5 directories, 5 filesregistry.home.labhosts.toml configdocker.iohosts.toml config_defaulthosts.toml configThis same setup is working with RKE2 and the mirror registry config with it's deployment.
Really throwing me for a loop. I'm sure I have something missing in the config just not sure what. I did try having
:5000at the end of the mirror registry but was getting the same response.The text was updated successfully, but these errors were encountered: