Skip to content

The reexec process does not read the rootfs related annnotations from the shim #816

Description

@cmainas

There is an issue with the way we pass the rootfs annotations from the shim to urunc. The problem is that the reexec process reads the annotations from the spec (config.json) file (See https://github.com/urunc-dev/urunc/blob/main/pkg/unikontainers/unikontainers.go#L128). However, the shim writes the rootfs related annotations in the spec after the task creation from the shim (See https://github.com/urunc-dev/urunc/blob/main/pkg/containerd-shim/task_service.go#L51) and therefore after urunc create runs. As a result, the reexec process never reads the annotations written from the shim.

To verify the above:
Use the below patch in urunc:

diff --git a/pkg/unikontainers/unikontainers.go b/pkg/unikontainers/unikontainers.go
index e226588..d9f6156 100644
--- a/pkg/unikontainers/unikontainers.go
+++ b/pkg/unikontainers/unikontainers.go
@@ -69,6 +69,7 @@ func New(bundlePath string, containerID string, rootDir string, cfg *UruncCon
fig
        if err != nil {
                return nil, err
        }
+       uniklog.Debugf("spec is %w", spec)
 
        if spec == nil || spec.Linux == nil {
                return nil, fmt.Errorf("invalid OCI spec: linux section is required")
@@ -132,9 +133,12 @@ func Get(containerID string, rootDir string) (*Unikontainer, error) {
        if spec == nil || spec.Linux == nil {
                return nil, fmt.Errorf("invalid OCI spec: linux section is required")
        }
+       uniklog.Debugf("spec in get is %w", spec)
        u.BaseDir = containerDir
        u.RootDir = rootDir
        u.Spec = spec
+       uniklog.Debugf("spec in get is %w", u.Spec)
+       uniklog.Debugf("annots in get is %w", u.Spec.Annotations)
        u.UruncCfg = UruncConfigFromMap(state.Annotations)
        return u, nil

Then execute any container that uses devmapper and check the urunc logs. The reexec process (with pid 1) does not have the rootfs annotations, while later urunc invocations (e.g. delete) has the annotations. For example:

Jul 10 14:51:20 cm-urunc-dev urunc[1]: {"file":"/home/cmainas/urunc/pkg/unikontainers/unikontai
ners.go:142","func":"github.com/urunc-dev/urunc/pkg/unikontainers.Get","level":"debug","msg":"a
nnots in get is %!w(map[string]string=map[com.urunc.unikernel.binary:/.boot/kernel com.urunc.un
ikernel.hypervisor:cloud-hypervisor com.urunc.unikernel.mountRootfs:true com.urunc.unikernel.un
ikernelType:linux nerdctl/auto-remove:true nerdctl/dns:{\"DNSServers\":null,\"DNSResolvConfOpti
ons\":null,\"DNSSearchDomains\":null} nerdctl/domainname: nerdctl/extraHosts:[] nerdctl/host-co
nfig:{\"BlkioWeight\":0,\"CidFile\":\"\",\"Devices\":null} nerdctl/hostname:d9ede7b32935 nerdct
l/ipc:{\"mode\":\"private\"} nerdctl/log-config:{\"driver\":\"json-file\",\"address\":\"/run/co
ntainerd/containerd.sock\"} nerdctl/log-uri:binary:///usr/local/bin/nerdctl?_NERDCTL_INTERNAL_L
OGGING=%2Fvar%2Flib%2Fnerdctl%2F1935db59 nerdctl/name:nginx-cloud-hypervisor-linux-raw-d9ede ne
rdctl/namespace:default nerdctl/networks:[\"bridge\"] nerdctl/platform:linux/amd64 nerdctl/stat
e-dir:/var/lib/nerdctl/1935db59/containers/default/d9ede7b32935a34cb93e191f6106cbb453a27364afbc
09ae2dc15ea7422db23b])","subsystem":"unikontainers","time":"2026-07-10T14:51:20Z"}
Jul 10 14:51:20 cm-urunc-dev urunc[1]: {"file":"/home/cmainas/urunc/pkg/unikontainers/unikontai
ners.go:936","func":"github.com/urunc-dev/urunc/pkg/unikontainers.(*Unikontainer).ExecuteHooks"
,"level":"debug","msg":"Executing 0 CreateContainer hooks","subsystem":"unikontainers","time":"
2026-07-10T14:51:20Z"}
Jul 10 14:51:20 cm-urunc-dev urunc[14718]: {"file":"/home/cmainas/urunc/pkg/unikontainers/uniko
ntainers.go:142","func":"github.com/urunc-dev/urunc/pkg/unikontainers.Get","level":"debug","msg
":"annots in get is %!w(map[string]string=map[com.urunc.internal.rootfs.params:{\"Type\":\"virt
iofs\",\"Path\":\"/run/containerd/io.containerd.runtime.v2.task/default/d9ede7b32935a34cb93e191
f6106cbb453a27364afbc09ae2dc15ea7422db23b/rootfs\",\"MountedPath\":\"/run/containerd/io.contain
erd.runtime.v2.task/default/d9ede7b32935a34cb93e191f6106cbb453a27364afbc09ae2dc15ea7422db23b/ro
otfs\",\"MonRootfs\":\"/run/containerd/io.containerd.runtime.v2.task/default/d9ede7b32935a34cb9
3e191f6106cbb453a27364afbc09ae2dc15ea7422db23b/monRootfs\"} com.urunc.unikernel.binary:/.boot/k
ernel com.urunc.unikernel.hypervisor:cloud-hypervisor com.urunc.unikernel.mountRootfs:true com.
urunc.unikernel.unikernelType:linux nerdctl/auto-remove:true nerdctl/dns:{\"DNSServers\":null,\
"DNSResolvConfOptions\":null,\"DNSSearchDomains\":null} nerdctl/domainname: nerdctl/extraHosts:
[] nerdctl/host-config:{\"BlkioWeight\":0,\"CidFile\":\"\",\"Devices\":null} nerdctl/hostname:d
9ede7b32935 nerdctl/ipc:{\"mode\":\"private\"} nerdctl/log-config:{\"driver\":\"json-file\",\"a
ddress\":\"/run/containerd/containerd.sock\"} nerdctl/log-uri:binary:///usr/local/bin/nerdctl?_
NERDCTL_INTERNAL_LOGGING=%2Fvar%2Flib%2Fnerdctl%2F1935db59 nerdctl/name:nginx-cloud-hypervisor-
linux-raw-d9ede nerdctl/namespace:default nerdctl/networks:[\"bridge\"] nerdctl/platform:linux/
amd64 nerdctl/state-dir:/var/lib/nerdctl/1935db59/containers/default/d9ede7b32935a34cb93e191f61
06cbb453a27364afbc09ae2dc15ea7422db23b])","subsystem":"unikontainers","time":"2026-07-10T14:51:
20Z"}
Jul 10 14:51:20 cm-urunc-dev urunc[14724]: {"file":"/home/cmainas/urunc/pkg/unikontainers/uniko
ntainers.go:142","func":"github.com/urunc-dev/urunc/pkg/unikontainers.Get","level":"debug","msg
":"annots in get is %!w(map[string]string=map[com.urunc.internal.rootfs.params:{\"Type\":\"virt
iofs\",\"Path\":\"/run/containerd/io.containerd.runtime.v2.task/default/d9ede7b32935a34cb93e191
f6106cbb453a27364afbc09ae2dc15ea7422db23b/rootfs\",\"MountedPath\":\"/run/containerd/io.contain
erd.runtime.v2.task/default/d9ede7b32935a34cb93e191f6106cbb453a27364afbc09ae2dc15ea7422db23b/ro
otfs\",\"MonRootfs\":\"/run/containerd/io.containerd.runtime.v2.task/default/d9ede7b32935a34cb9
3e191f6106cbb453a27364afbc09ae2dc15ea7422db23b/monRootfs\"} com.urunc.unikernel.binary:/.boot/k
ernel com.urunc.unikernel.hypervisor:cloud-hypervisor com.urunc.unikernel.mountRootfs:true com.
urunc.unikernel.unikernelType:linux nerdctl/auto-remove:true nerdctl/dns:{\"DNSServers\":null,\
"DNSResolvConfOptions\":null,\"DNSSearchDomains\":null} nerdctl/domainname: nerdctl/extraHosts:
[] nerdctl/host-config:{\"BlkioWeight\":0,\"CidFile\":\"\",\"Devices\":null} nerdctl/hostname:d9ede7b32935 nerdctl/ipc:{\"mode\":\"private\"} nerdctl/log-config:{\"driver\":\"json-file\",\"address\":\"/run/containerd/containerd.sock\"} nerdctl/log-uri:binary:///usr/local/bin/nerdctl?_NERDCTL_INTERNAL_LOGGING=%2Fvar%2Flib%2Fnerdctl%2F1935db59 nerdctl/name:nginx-cloud-hypervisor-linux-raw-d9ede nerdctl/namespace:default nerdctl/networks:[\"bridge\"] nerdctl/platform:linux/amd64 nerdctl/state-dir:/var/lib/nerdctl/1935db59/containers/default/d9ede7b32935a34cb93e191f6106cbb453a27364afbc09ae2dc15ea7422db23b])","subsystem":"unikontainers","time":"2026-07-10T14:51:20Z"}

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Fields

No fields configured for Bug.

Projects

Status
No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions