Failed to start ostree-boot-complete.service - OSTree Complete Boot.

[jtligon]

Describe the bug

migrated my fedoraiot system over to bootc
containerfile here https://github.com/jtligon/silver-octo-dollop/blob/main/fitlet.Containerfile
journalctl here

Aug 28 16:05:36 fitlet2 ostree[862]: error: ostree-finalize-staged.service failed on previous boot: Bootloader write config: grub2-mkconfig: Child process exited with code 1
Aug 28 16:05:36 fitlet2 systemd[1]: ostree-boot-complete.service: Main process exited, code=exited, status=1/FAILURE
Aug 28 16:05:36 fitlet2 systemd[1]: ostree-boot-complete.service: Failed with result 'exit-code'.
Aug 28 16:05:36 fitlet2 systemd[1]: Failed to start ostree-boot-complete.service - OSTree Complete Boot.
Aug 28 16:17:43 fitlet2 systemd[1]: ostree-boot-complete.service - OSTree Complete Boot was skipped because no trigger condition checks were met.
Aug 28 16:19:40 fitlet2 systemd[1]: ostree-boot-complete.service - OSTree Complete Boot was skipped because no trigger condition checks were met.

Reproduction steps

1. rebase to bootc
2. rpm-ostree reset to get rid of previous layered packages
3. reboot

Expected behavior

no failed systemd units

Actual behavior

[systemd]
Failed Units: 1
  ostree-boot-complete.service

on login

System details

rpm-ostree:
 Version: '2024.7'
 Git: 21911f1510b44d8962b28ff86a92f789f1db0095
 Features:
  - rust
  - compose
  - container
  - fedora-integration

Additional information

full journal here https://paste.centos.org/view/d29087af

[miabbott]

From the journal, I think the real error is:

Aug 28 16:05:36 fitlet2 ostree[862]: error: ostree-finalize-staged.service failed on previous boot: Bootloader write config: grub2-mkconfig: Child process exited with code 1

Since the failure is in the previous boot, could you provide the journal for the previous boot where ostree-finalize-staged failed? You can use journalctl -b -1 to retrieve it.

[jtligon]

is 60mb big for a log file? seems big...

I cut 58 MB out of the middle for this file
fitlet2_journal-1.txt

[miabbott]

Aug 28 16:04:38 fitlet2 ostree[739449]: Finalizing staged deployment
Aug 28 16:04:40 fitlet2 ostree[739449]: Copying /etc changes: 19 modified, 1 removed, 56 added
Aug 28 16:04:40 fitlet2 ostree[739449]: Copying /etc changes: 19 modified, 1 removed, 56 added
Aug 28 16:04:42 fitlet2 ostree[739449]: Refreshing SELinux policy
Aug 28 16:04:54 fitlet2 ostree[739449]: Refreshed SELinux policy in 11532 ms
Aug 28 16:04:54 fitlet2 ostree[739449]: Finalized deployment
Aug 28 16:04:54 fitlet2 ostree[739449]: bootfs is sufficient for calculated new size: 0 bytes
Aug 28 16:04:54 fitlet2 ostree[739449]: Regex version mismatch, expected: 10.44 2024-06-07 actual: 10.42 2022-12-11
Aug 28 16:04:54 fitlet2 ostree[739449]: Regex version mismatch, expected: 10.44 2024-06-07 actual: 10.42 2022-12-11
Aug 28 16:04:55 fitlet2 ostree[739449]: error: Bootloader write config: grub2-mkconfig: Child process exited with code 1
Aug 28 16:04:55 fitlet2 systemd[1]: ostree-finalize-staged.service: Control process exited, code=exited, status=1/FAILURE
Aug 28 16:04:55 fitlet2 systemd[1]: ostree-finalize-staged.service: Failed with result 'exit-code'.

It's not obvious to me from that journal why writing the bootloader failed.

Though bootfs is sufficient for calculated new size: 0 bytes feels odd and may be a problem

(I see ostree[40307]: bootfs is sufficient for calculated new size: 158.8 MB on my Silverblue host)

[jtligon]

[jtligon]

anedata: I remember rolling back on this device when my previous bootc attempts were missing a wifi driver. after I did rollback, it failed to write the new grub config... no idea why.

[cgwalters]

Oh ugh, we suppress stderr from grub2-mkconfig by default...

[cgwalters]

ostreedev/ostree#3290

But, what happens when you just run grub2-mkconfig -o /dev/null?

[jtligon]

you must run this as root...

i kid i kid

/usr/sbin/grub2-probe: error: failed to get canonical path of overlay'.`

[cgwalters]

Yeah as I expected it's the same as ostreedev/ostree#3198 ...this legacy of silverblue and IoT using dynamic grub configs is so extremely painful...

[cgwalters]

Just to save you lots of cross-digging through bugs, I filed a patch in https://bugzilla.redhat.com/show_bug.cgi?id=2308594

You can test it out with e.g. bootc usroverlay and applying that live