krunkit: secondary "disk:" volume mounts on first boot but breaks ("Unexpected disk at mount point.") on every subsequent restart

[carlossg]

Description

On a krunkit vmType profile, the secondary data disk (the one sized by disk: in colima.yaml, as opposed to rootDisk:) mounts correctly on the VM's first boot, but is silently left unmounted after any subsequent plain colima stop / colima start cycle — no config change, no delete/recreate needed. provision.dependency/00000007 (disk.sh) fails on that second boot with "Unexpected disk at mount point.", and a separate root-disk resize step (provision.system/00000006) also starts failing at the same time with resize2fs: Please run 'e2fsck -f ...' first, pointed at the very disk that disk.sh is trying (and failing) to mount. Both failures make cloud-final.service exit with status 1 (failed).

Net effect: container/model data (/var/lib/docker, /var/lib/containerd, /var/lib/ramalama, etc.) silently falls back onto the small rootDisk (default 20GiB) instead of the large disk: volume after the first restart, so the VM runs out of space almost immediately when pulling large images/models (e.g. colima model serve --runner ramalama <30B model> fails with "no space left on device").

Reproduced twice independently: once on a long-lived profile (krunkit, delete + recreate did not help), and once from scratch on a brand-new profile created solely to isolate this (krunkit-repro) — full commands and output below. The equivalent vz profile (default, macOS Virtualization.framework) mounts its secondary disk correctly on every boot and has no such issue.

I checked main at HEAD (past the v0.10.3 tag): neither environment/vm/lima/disk.sh nor environment/vm/lima/yaml.go (which owns the resize2fs/growpart root-disk logic) has any commits since v0.10.3, so this is still present on current main, not just the released version.

Likely related to #1570/#1571 (fixed the "already mounted" detection in disk.sh using findmnt, present in my version) — but a different branch of the same script (the second findmnt --target guard, "Unexpected disk at mount point.") is the one failing here. Also very likely the underlying cause of #1583 ("krunkit vm data lost on restart"), which reports the same "works once, breaks on restart" pattern.

Separately, note for maintainers: while reproducing this on a fresh profile I also hit #1607 (mountType must be virtiofs or reverse-sshfs, got 9p) on plain colima start -p <profile> --vm-type krunkit. That one is already fixed on main (#1609, merged after v0.10.3) but not yet in a tagged release — mentioning it only so it isn't confused with this report; I worked around it with --mount-type sshfs to get to the actual repro below.

Version

colima version 0.10.3
git commit: 00f6c297e92a82c04a4ab507db0a61435650d7e8

runtime: docker
arch: aarch64
client: v28.5.1
server: v29.5.2
limactl version 2.2.0
qemu-img version 11.0.3

krunkit 1.2.2

Operating System

• macOS Intel <= 13 (Ventura)
• macOS Intel >= 14 (Sonoma)
• Apple Silicon >= 14 (Sonoma) — macOS 26.6 (25G72)
• Linux

Reproduction Steps (clean, isolated profile)

$ colima start -p krunkit-repro --vm-type krunkit --disk 40
...
level=error msg="Failed to set config" error="rpc error: code = Unknown desc = field `mountType` must be `virtiofs` or `reverse-sshfs` for krunkit driver, got `9p`"

(→ this is #1607, fixed on main via #1609 but not yet released; worked around below.)

$ colima delete -p krunkit-repro -y
$ colima start -p krunkit-repro --vm-type krunkit --mount-type sshfs --disk 40
... (starts fine)

First boot — disk mounts correctly:

$ colima ssh -p krunkit-repro -- lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda     253:0    0   20G  0 disk
├─vda1  253:1    0   19G  0 part /
├─vda15 253:15   0   99M  0 part /boot/efi
└─vda16 259:0    0  923M  0 part /boot
vdb     253:16   0   19M  0 disk /mnt/lima-cidata
vdc     253:32   0   40G  0 disk
└─vdc1  253:33   0   40G  0 part /var/lib/ramalama
                                 /var/lib/cni
                                 /var/lib/rancher
                                 /var/lib/containerd
                                 /var/lib/docker
                                 /mnt/lima-colima-krunkit-repro

$ colima ssh -p krunkit-repro -- systemctl is-active cloud-final.service
failed   # (this boot's failure was an unrelated transient DNS-not-ready issue during
         #  apt-get install rsync in boot.Linux/30-install-packages.sh — disk itself
         #  mounted fine, included for completeness)

Restart with zero config changes — disk comes back unmounted:

$ colima stop -p krunkit-repro
$ colima start -p krunkit-repro
$ colima ssh -p krunkit-repro -- lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda     253:0    0   20G  0 disk
├─vda1  253:1    0   19G  0 part /var/lib/ramalama
│                                /var/lib/cni
│                                /var/lib/rancher
│                                /var/lib/containerd
│                                /var/lib/docker
│                                /
├─vda15 253:15   0   99M  0 part /boot/efi
└─vda16 259:0    0  923M  0 part /boot
vdb     253:16   0   19M  0 disk /mnt/lima-cidata
vdc     253:32   0 39.9G  0 disk
└─vdc1  253:33   0 39.9G  0 part          # <- no mountpoint anymore

$ colima ssh -p krunkit-repro -- systemctl is-active cloud-final.service
failed

Journal for that failing boot:

$ colima ssh -p krunkit-repro -- sudo journalctl -b -u cloud-final.service -o cat
...
+ [ -d /mnt/lima-cidata/provision.dependency ]
+ CODE=0
+ /mnt/lima-cidata/provision.dependency/00000001
+ /mnt/lima-cidata/provision.dependency/00000007
/dev/vdb         20M   20M     0 100% /mnt/lima-cidata
Unexpected disk at mount point.
+ CODE=1
+ /mnt/lima-cidata/provision.dependency/00000008
...
+ [ 1 != 0 ]
+ exit 1
LIMA ...| WARNING: Failed to execute /mnt/lima-cidata/boot.Linux/30-install-packages.sh
...
LIMA ...| Executing /mnt/lima-cidata/provision.system/00000006
resize2fs 1.47.0 (5-Feb-2023)
Please run 'e2fsck -f /dev/disk/by-label/lima-colima-krun' first.
LIMA ...| Exiting with code 1
...
cloud-final.service: Main process exited, code=exited, status=1/FAILURE
cloud-final.service: Failed with result 'exit-code'.
Failed to start cloud-final.service - Cloud-init: Final Stage.

blkid confirms /dev/disk/by-label/lima-colima-krun unambiguously resolves to /dev/vdc1 (the secondary disk; the root disk is separately labelled cloudimg-rootfs), so provision.system/00000006's resize2fs step is being run against the secondary disk, which apparently didn't get cleanly unmounted at colima stop and now needs e2fsck before it can be touched — which then aborts the whole provisioning chain with set -e-style propagation, so disk.sh's mount attempt right before it (00000007) never gets a clean run either.

$ colima ssh -p krunkit-repro -- sudo blkid
/dev/vdb: BLOCK_SIZE="2048" UUID="..." LABEL="cidata" TYPE="iso9660"
/dev/vdc1: LABEL="lima-colima-krun" UUID="..." BLOCK_SIZE="4096" TYPE="ext4"
/dev/vda15: LABEL_FATBOOT="UEFI" LABEL="UEFI" ...
/dev/vda1: LABEL="cloudimg-rootfs" UUID="..." BLOCK_SIZE="4096" TYPE="ext4" ...
/dev/vda16: LABEL="BOOT" UUID="..." ...

For comparison, the default profile (vmType: vz) on the same host mounts its equivalent disk fine on every boot:

$ colima ssh -p default -- df -h
...
/dev/vdb1              59G   36G   21G  64% /mnt/lima-colima
$ colima ssh -p default -- systemctl is-active cloud-final.service
active

Expected behaviour

• disk.sh mounts the secondary disk on every boot on krunkit, not just the first.
• The root-disk resize step (provision.system/00000006) never targets the secondary disk.
• cloud-final.service completes successfully (active) after a normal stop/start, not failed.
• Container/model data stays backed by the large disk: volume rather than silently falling back to the small rootDisk after a restart.

Workaround

Bumping rootDisk instead of relying on disk: works around the space problem (though cloud-final.service still ends up failed for the reasons above):

colima stop -p krunkit
colima start -p krunkit --root-disk 100

vda1 grows in place via colima's own growpart/resize2fs and everything under /var/lib gets the extra space, independent of the broken second-disk mount.

Additional context

• Confirmed present on current main (past v0.10.3): no commits to environment/vm/lima/disk.sh or environment/vm/lima/yaml.go since the v0.10.3 tag.
• Ruled out the fstrim/discard bug in [Bug]: provision.system/00000005 fails on krunkit (macOS) due to unsupported fstrim #1549 (provision.system/00000005) as the cause here — manually re-running that script inside the VM exits 0 cleanly on this image.
• Ruled out ext4 16-char label truncation/collision between root and secondary disk as the cause — blkid shows distinct labels (cloudimg-rootfs vs lima-colima-krun).
• Happy to provide the full raw journalctl -b -u cloud-final.service output (trimmed above for brevity) or run further diagnostics.