Empty or invalid `.../containerd/daemon/io.containerd.grpc.v1.introspection/uuid` file created occasionally

[lmbarros]

In some rare cases, the /mnt/data/docker/containerd/daemon/io.containerd.grpc.v1.introspection/uuid created by containerd is empty (instead of containing an UUID). This causes some containerd operations to fail. Notably, the following command (which happens to be part of the current balenaEngine health check)

# balena-engine-containerd-ctr --address "/run/balena-engine/containerd/balena-engine-containerd.sock" version

will fail with

invalid UUID length: 0: unknown

[jellyfish-bot]

[lmbarros] This has attached https://jel.ly.fish/e28ad781-ab1e-459d-b017-935900ccd76c

[lmbarros]

Saw a variation of this today, in two different devices. In both cases the uuid file was created, but with invalid contents. Interestingly, in both cases the contents of the file were the same:

# hexdump /mnt/data/docker/containerd/daemon/io.containerd.grpc.v1.introspection/uuid 
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
0000020 0000 0000                              
0000024

# sha1sum /mnt/data/docker/containerd/daemon/io.containerd.grpc.v1.introspection/uuid 
8696cf0f4655636cc93c566c1be2dad311da646c  /mnt/data/docker/containerd/daemon/io.containerd.grpc.v1.introspection/uuid

[alanb128]

Answers to questions posed about the devices referenced above, where the UUID file was created but with invalid contents:

Do you know when these two devices were fist provisioned?

According to Balena dashboard they were created Nov 2022 and Jan 2023. It should reflect real commission date. Many other devices from this fleet were commissioned in the same time interval.

From what I see in the logs, all the OS update attempts on them were made in the last 2 or 3 days. Can you confirm that, or there were update attempts (successful or not) from before that?

I cannot be 100% sure as we do not track them, but I would say that any OS updates attempts before that are very unlikely. But there probably were a supervisor update to 14.13.14 some time before mentioned OS update.

Did you ever notice anything wrong with these devices before? Do you think they could have been in this state (containers restarting every 6 minutes) for long without you noticing?

I noticed repeated supervisor restarts only just before running OS updates. I was hoping that OS update could fix it as well.

[lmbarros]

Same user reported a device with the same symptoms (same invalid contents in the uuid file) on device running balenaOS v3.1.1 and that hasn't gone though OS or Supervisor updates ever. And according to stat and our database, the uuid file was created when the first provisioned and never changed since then. This should rule out any relationship of the issue with HUPs.