Skip to content

Add Fedora CoreOS to CI tests - #1806

Open
Rolv-Apneseth wants to merge 2 commits into
containers:mainfrom
Rolv-Apneseth:fcos_ci
Open

Add Fedora CoreOS to CI tests#1806
Rolv-Apneseth wants to merge 2 commits into
containers:mainfrom
Rolv-Apneseth:fcos_ci

Conversation

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor

Closes #714, which also includes the context for this PR.

This is also testing whether the images added to the SF nodepool in https://gitlab.com/softwarefactory-project/config/-/merge_requests/63#note_3445923313 actually work as intended so there may be some experimentation.

Let me know if there are any obvious issues, but otherwise I will let you know if/when tests are passing @debarshiray.

Assisted-by: Claude (claude-opus-4-6)

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces Fedora CoreOS system test jobs in Zuul and refactors the dependency playbooks by extracting shared tasks into a new common playbook (dependencies-common.yaml). Feedback on the changes highlights that the rpm-ostree install command in dependencies-fedora-coreos.yaml uses an unsupported --idempotent flag which will cause failures, and recommends adding the missing flatpak-session-helper package to prevent potential failures in Toolbx system tests.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread playbooks/dependencies-fedora-coreos.yaml
Comment thread playbooks/dependencies-fedora-coreos.yaml
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/1e96f57e18e44daba4a1ddd9ee594b87

✔️ unit-test SUCCESS in 2m 27s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 43s
✔️ unit-test-restricted SUCCESS in 2m 30s
system-test-fedora-rawhide-commands-options TIMED_OUT in 2h 00m 26s
✔️ system-test-fedora-rawhide-runtime-environment-arch-fedora SUCCESS in 28m 27s
✔️ system-test-fedora-rawhide-runtime-environment-ubuntu SUCCESS in 7m 18s
system-test-fedora-44-commands-options TIMED_OUT in 1h 45m 32s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 22m 36s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 6m 17s
✔️ system-test-fedora-43-commands-options SUCCESS in 32m 12s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 22m 50s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 6m 33s
✔️ system-test-fedora-42-commands-options SUCCESS in 32m 20s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 23m 01s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 6m 31s
system-test-fedora-coreos-next-commands-options NODE_FAILURE Node(set) request 200-0000117118 failed in 0s
system-test-fedora-coreos-next-runtime-environment-arch-fedora NODE_FAILURE Node(set) request 200-0000117119 failed in 0s
system-test-fedora-coreos-next-runtime-environment-ubuntu NODE_FAILURE Node(set) request 200-0000117120 failed in 0s
system-test-fedora-coreos-stable-commands-options NODE_FAILURE Node(set) request 200-0000117121 failed in 0s
system-test-fedora-coreos-stable-runtime-environment-arch-fedora NODE_FAILURE Node(set) request 200-0000117122 failed in 0s
system-test-fedora-coreos-stable-runtime-environment-ubuntu NODE_FAILURE Node(set) request 200-0000117123 failed in 0s
system-test-fedora-coreos-testing-commands-options NODE_FAILURE Node(set) request 200-0000117124 failed in 0s
system-test-fedora-coreos-testing-runtime-environment-arch-fedora NODE_FAILURE Node(set) request 200-0000117125 failed in 0s
system-test-fedora-coreos-testing-runtime-environment-ubuntu NODE_FAILURE Node(set) request 200-0000117126 failed in 0s

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/1f88b41228134dda9e3663f26efcdca3

✔️ unit-test SUCCESS in 2m 21s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 02s
✔️ unit-test-restricted SUCCESS in 2m 17s
system-test-fedora-rawhide-commands-options TIMED_OUT in 2h 00m 32s
✔️ system-test-fedora-rawhide-runtime-environment-arch-fedora SUCCESS in 28m 38s
✔️ system-test-fedora-rawhide-runtime-environment-ubuntu SUCCESS in 7m 27s
system-test-fedora-44-commands-options TIMED_OUT in 1h 45m 28s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 22m 38s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 6m 33s
system-test-fedora-43-commands-options TIMED_OUT in 1h 45m 32s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 22m 56s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 6m 30s
✔️ system-test-fedora-42-commands-options SUCCESS in 32m 27s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 23m 22s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 6m 15s
system-test-fedora-coreos-next-commands-options NODE_FAILURE Node(set) request 200-0000117669 failed in 0s
system-test-fedora-coreos-next-runtime-environment-arch-fedora NODE_FAILURE Node(set) request 200-0000117670 failed in 0s
system-test-fedora-coreos-next-runtime-environment-ubuntu NODE_FAILURE Node(set) request 200-0000117671 failed in 0s
system-test-fedora-coreos-stable-commands-options NODE_FAILURE Node(set) request 200-0000117672 failed in 0s
system-test-fedora-coreos-stable-runtime-environment-arch-fedora NODE_FAILURE Node(set) request 200-0000117673 failed in 0s
system-test-fedora-coreos-stable-runtime-environment-ubuntu NODE_FAILURE Node(set) request 200-0000117674 failed in 0s
system-test-fedora-coreos-testing-commands-options NODE_FAILURE Node(set) request 200-0000117675 failed in 0s
system-test-fedora-coreos-testing-runtime-environment-arch-fedora NODE_FAILURE Node(set) request 200-0000117676 failed in 0s
system-test-fedora-coreos-testing-runtime-environment-ubuntu NODE_FAILURE Node(set) request 200-0000117677 failed in 0s

@debarshiray debarshiray left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for soldiering on with the Fedora CoreOS tests, @Rolv-Apneseth ! This is exciting.

It seems like we can land the first commit independently from the rest of the work and the CI turning green. It seems useful on its own, and it will keep things moving. Is it sufficiently well settled already?

Comment thread playbooks/setup-env.yaml
Comment thread .zuul.yaml
@Rolv-Apneseth

Copy link
Copy Markdown
Contributor Author

Thanks for soldiering on with the Fedora CoreOS tests, @Rolv-Apneseth ! This is exciting.

No worries! It will be cool to finally have this done.

It seems like we can land the first commit independently from the rest of the work and the CI turning green. It seems useful on its own, and it will keep things moving. Is it sufficiently well settled already?

Sure, would you want it split off into a separate PR? Still working with the SF maintainers to get the CoreOS nodes working so this PR may take a while longer.

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor Author

By the way, are those timeouts in the commands-options runs common / known to happen?

@Rolv-Apneseth
Rolv-Apneseth force-pushed the fcos_ci branch 2 times, most recently from 88ee98e to 6d0f743 Compare June 16, 2026 09:01
@Rolv-Apneseth

Copy link
Copy Markdown
Contributor Author

Looks like the next and testing nodes are passing now. The jobs on stable nodes fail due to a version mismatch when trying to layer in glibc-devel, since the glib shipped with the image is an older version no longer available in the repos it's pulling from. I'll try adding a system update step to avoid any issues like that, which I believe can happen on any stream.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/ff3bd316f0ec4a6d890297f417ad1763

✔️ unit-test SUCCESS in 2m 28s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 17s
✔️ unit-test-restricted SUCCESS in 2m 27s
✔️ system-test-fedora-rawhide-commands-options SUCCESS in 40m 29s
✔️ system-test-fedora-rawhide-runtime-environment-arch-fedora SUCCESS in 28m 32s
✔️ system-test-fedora-rawhide-runtime-environment-ubuntu SUCCESS in 7m 56s
system-test-fedora-44-commands-options TIMED_OUT in 1h 45m 26s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 23m 29s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 6m 59s
system-test-fedora-43-commands-options TIMED_OUT in 1h 45m 30s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 23m 32s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 7m 01s
✔️ system-test-fedora-42-commands-options SUCCESS in 33m 00s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 23m 58s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 6m 36s
system-test-fedora-coreos-next-commands-options TIMED_OUT in 1h 45m 35s (non-voting)
system-test-fedora-coreos-next-runtime-environment-arch-fedora FAILURE in 30m 15s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-ubuntu SUCCESS in 9m 32s (non-voting)
system-test-fedora-coreos-stable-commands-options RETRY_LIMIT in 56s (non-voting)
system-test-fedora-coreos-stable-runtime-environment-arch-fedora RETRY_LIMIT in 1m 07s (non-voting)
system-test-fedora-coreos-stable-runtime-environment-ubuntu RETRY_LIMIT in 58s (non-voting)
system-test-fedora-coreos-testing-commands-options TIMED_OUT in 1h 45m 28s (non-voting)
system-test-fedora-coreos-testing-runtime-environment-arch-fedora FAILURE in 30m 32s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-ubuntu SUCCESS in 9m 20s (non-voting)

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor Author

So the updating didn't work since it only updates to the latest available image (which it already is in this case). I'll keep that step anyway as it's good to avoid similar issues when the node image is older than the latest available one. I'll try see if there's any other workaround we can take here.

The failures in testing-runtime-environment-arch-fedora were due to:

not ok 88 environment variables: HOSTNAME inside Fedora 34
# tags: arch-fedora runtime-environment
# (from function `assert_line' in file test/system/libs/bats-assert/src/assert.bash, line 488,
#  in test file test/system/220-environment-variables.bats, line 406)
#   `assert_line --index 0 "$HOSTNAME"' failed
#
# -- line differs --
# index    : 0
# expected : localhost.localdomain
# actual   : localhost
# --
#

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/94c994ab5d984a16a4f26e611c0c6643

✔️ unit-test SUCCESS in 2m 26s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 17s
✔️ unit-test-restricted SUCCESS in 2m 35s
✔️ system-test-fedora-rawhide-commands-options SUCCESS in 41m 57s
✔️ system-test-fedora-rawhide-runtime-environment-arch-fedora SUCCESS in 28m 14s
✔️ system-test-fedora-rawhide-runtime-environment-ubuntu SUCCESS in 7m 24s
system-test-fedora-44-commands-options TIMED_OUT in 1h 45m 33s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 25m 07s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 7m 29s
system-test-fedora-43-commands-options TIMED_OUT in 1h 45m 32s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 25m 08s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 6m 44s
✔️ system-test-fedora-42-commands-options SUCCESS in 33m 51s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 23m 28s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 7m 25s
system-test-fedora-coreos-next-commands-options TIMED_OUT in 1h 45m 36s (non-voting)
system-test-fedora-coreos-next-runtime-environment-arch-fedora FAILURE in 31m 15s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-ubuntu SUCCESS in 10m 34s (non-voting)
system-test-fedora-coreos-stable-commands-options RETRY_LIMIT in 1m 05s (non-voting)
system-test-fedora-coreos-stable-runtime-environment-arch-fedora RETRY_LIMIT in 1m 03s (non-voting)
system-test-fedora-coreos-stable-runtime-environment-ubuntu RETRY_LIMIT in 1m 00s (non-voting)
system-test-fedora-coreos-testing-commands-options TIMED_OUT in 1h 45m 33s (non-voting)
system-test-fedora-coreos-testing-runtime-environment-arch-fedora FAILURE in 30m 59s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-ubuntu SUCCESS in 10m 24s (non-voting)

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/537255fa2bba43dbae8079afbb2a4d3d

✔️ unit-test SUCCESS in 2m 25s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 06s
✔️ unit-test-restricted SUCCESS in 2m 29s
✔️ system-test-fedora-rawhide-commands-options SUCCESS in 41m 44s
✔️ system-test-fedora-rawhide-runtime-environment-arch-fedora SUCCESS in 29m 07s
✔️ system-test-fedora-rawhide-runtime-environment-ubuntu SUCCESS in 7m 25s
system-test-fedora-44-commands-options TIMED_OUT in 1h 45m 26s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 24m 32s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 6m 30s
system-test-fedora-43-commands-options TIMED_OUT in 1h 45m 28s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 23m 05s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 6m 30s
✔️ system-test-fedora-42-commands-options SUCCESS in 34m 22s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 23m 30s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 6m 36s
system-test-fedora-coreos-next-commands-options TIMED_OUT in 1h 45m 35s (non-voting)
system-test-fedora-coreos-next-runtime-environment-arch-fedora FAILURE in 33m 48s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-ubuntu SUCCESS in 10m 17s (non-voting)
✔️ system-test-fedora-coreos-stable-commands-options SUCCESS in 42m 47s (non-voting)
system-test-fedora-coreos-stable-runtime-environment-arch-fedora FAILURE in 34m 08s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-ubuntu SUCCESS in 10m 57s (non-voting)
system-test-fedora-coreos-testing-commands-options TIMED_OUT in 1h 45m 37s (non-voting)
system-test-fedora-coreos-testing-runtime-environment-arch-fedora FAILURE in 33m 43s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-ubuntu SUCCESS in 10m 37s (non-voting)

@Rolv-Apneseth

Rolv-Apneseth commented Jun 16, 2026

Copy link
Copy Markdown
Contributor Author

The failure described in the previous comment probably needs to be investigated separately, maybe a separate issue?

As for the timeouts - any suggestions? @debarshiray

Edit: started looking into it but I see this is being investigated in #1805

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/a38a86d35774476f9b774b88128c5d5f

✔️ unit-test SUCCESS in 2m 24s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 13s
✔️ unit-test-restricted SUCCESS in 2m 54s
✔️ system-test-fedora-rawhide-commands-options SUCCESS in 52m 48s
✔️ system-test-fedora-rawhide-runtime-environment-arch-fedora SUCCESS in 29m 04s
✔️ system-test-fedora-rawhide-runtime-environment-ubuntu SUCCESS in 7m 27s
system-test-fedora-44-commands-options TIMED_OUT in 1h 45m 32s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 23m 53s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 7m 10s
system-test-fedora-43-commands-options TIMED_OUT in 1h 45m 35s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 22m 49s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 6m 32s
✔️ system-test-fedora-42-commands-options SUCCESS in 46m 32s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 23m 13s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 7m 15s
system-test-fedora-coreos-next-commands-options TIMED_OUT in 2h 00m 29s (non-voting)
system-test-fedora-coreos-next-runtime-environment-arch-fedora FAILURE in 42m 44s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-ubuntu SUCCESS in 13m 51s (non-voting)
✔️ system-test-fedora-coreos-stable-commands-options SUCCESS in 52m 01s (non-voting)
system-test-fedora-coreos-stable-runtime-environment-arch-fedora FAILURE in 42m 52s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-ubuntu SUCCESS in 13m 54s (non-voting)
system-test-fedora-coreos-testing-commands-options TIMED_OUT in 2h 00m 32s (non-voting)
system-test-fedora-coreos-testing-runtime-environment-arch-fedora FAILURE in 42m 43s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-ubuntu SUCCESS in 13m 47s (non-voting)

@Rolv-Apneseth
Rolv-Apneseth force-pushed the fcos_ci branch 3 times, most recently from ffe3d61 to 1063903 Compare June 18, 2026 14:45
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/a02028cfa06a477baa13f9ddd987d56b

✔️ unit-test SUCCESS in 2m 24s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 01s
✔️ unit-test-restricted SUCCESS in 2m 28s
✔️ system-test-fedora-rawhide-commands-options SUCCESS in 40m 43s
✔️ system-test-fedora-rawhide-runtime-environment-arch-fedora SUCCESS in 39m 45s
✔️ system-test-fedora-rawhide-runtime-environment-ubuntu SUCCESS in 7m 25s
✔️ system-test-fedora-44-commands-options SUCCESS in 31m 30s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 24m 12s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 6m 56s
system-test-fedora-43-commands-options TIMED_OUT in 1h 45m 31s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 23m 58s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 6m 19s
✔️ system-test-fedora-42-commands-options SUCCESS in 47m 00s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 24m 40s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 10m 49s
system-test-fedora-coreos-next-commands-options TIMED_OUT in 1h 45m 32s (non-voting)
system-test-fedora-coreos-next-runtime-environment-arch-fedora FAILURE in 42m 39s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-ubuntu SUCCESS in 14m 13s (non-voting)
✔️ system-test-fedora-coreos-stable-commands-options SUCCESS in 51m 43s (non-voting)
system-test-fedora-coreos-stable-runtime-environment-arch-fedora FAILURE in 42m 35s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-ubuntu SUCCESS in 14m 03s (non-voting)
system-test-fedora-coreos-testing-commands-options TIMED_OUT in 1h 45m 29s (non-voting)
system-test-fedora-coreos-testing-runtime-environment-arch-fedora FAILURE in 42m 42s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-ubuntu SUCCESS in 14m 06s (non-voting)

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor Author

Hostname issue is actually on the Zuul node side of things, opened another PR with SF to try and resolve that. Should be the final step here.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/f6122283c02c45578b5b945730a1d38f

unit-test RETRY_LIMIT in 27s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 22s
unit-test-restricted RETRY_LIMIT in 30s
system-test-fedora-rawhide-commands-options RETRY_LIMIT in 29s
system-test-fedora-rawhide-runtime-environment-arch-fedora RETRY_LIMIT in 29s
system-test-fedora-rawhide-runtime-environment-ubuntu RETRY_LIMIT in 30s
✔️ system-test-fedora-44-commands-options SUCCESS in 34m 53s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 22m 22s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 7m 26s
system-test-fedora-43-commands-options TIMED_OUT in 1h 45m 32s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 25m 22s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 7m 16s
✔️ system-test-fedora-42-commands-options SUCCESS in 32m 28s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 26m 06s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 7m 33s
✔️ system-test-fedora-coreos-next-commands-options SUCCESS in 42m 33s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-arch-fedora SUCCESS in 31m 25s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-ubuntu SUCCESS in 11m 20s (non-voting)
system-test-fedora-coreos-stable-commands-options TIMED_OUT in 1h 45m 29s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-arch-fedora SUCCESS in 31m 54s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-ubuntu SUCCESS in 11m 55s (non-voting)
✔️ system-test-fedora-coreos-testing-commands-options SUCCESS in 42m 36s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-arch-fedora SUCCESS in 33m 10s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-ubuntu SUCCESS in 11m 31s (non-voting)

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor Author

Hostname checks pass now, but re-running to see if we can get the ones that hit the retry limit to pass (the non-rawhide ones at least).

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/2670e89ab915484da87510d85eefd1ba

unit-test RETRY_LIMIT in 27s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 08s
unit-test-restricted RETRY_LIMIT in 28s
system-test-fedora-rawhide-commands-options RETRY_LIMIT in 27s
system-test-fedora-rawhide-runtime-environment-arch-fedora RETRY_LIMIT in 28s
system-test-fedora-rawhide-runtime-environment-ubuntu RETRY_LIMIT in 29s
✔️ system-test-fedora-44-commands-options SUCCESS in 43m 03s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 33m 09s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 6m 18s
system-test-fedora-43-commands-options TIMED_OUT in 1h 45m 35s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 23m 23s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 8m 35s
✔️ system-test-fedora-42-commands-options SUCCESS in 32m 02s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 24m 05s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 6m 26s
✔️ system-test-fedora-coreos-next-commands-options SUCCESS in 48m 15s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-arch-fedora SUCCESS in 40m 03s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-ubuntu SUCCESS in 11m 02s (non-voting)
system-test-fedora-coreos-stable-commands-options TIMED_OUT in 1h 45m 30s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-arch-fedora SUCCESS in 40m 05s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-ubuntu SUCCESS in 9m 45s (non-voting)
✔️ system-test-fedora-coreos-testing-commands-options SUCCESS in 39m 13s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-arch-fedora SUCCESS in 40m 10s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-ubuntu SUCCESS in 9m 46s (non-voting)

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor Author

Rebased after #1824

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/d88d74157bf74e71b6625a1acaaa8224

✔️ unit-test SUCCESS in 2m 32s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 28s
✔️ unit-test-restricted SUCCESS in 2m 24s
✔️ system-test-fedora-rawhide-commands-options SUCCESS in 53m 29s
✔️ system-test-fedora-rawhide-runtime-environment-arch-fedora SUCCESS in 39m 46s
✔️ system-test-fedora-rawhide-runtime-environment-ubuntu SUCCESS in 8m 57s
✔️ system-test-fedora-44-commands-options SUCCESS in 43m 42s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 32m 40s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 8m 13s
✔️ system-test-fedora-43-commands-options SUCCESS in 44m 08s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 32m 46s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 10m 13s
✔️ system-test-fedora-42-commands-options SUCCESS in 32m 24s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 27m 27s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 7m 24s
✔️ system-test-fedora-coreos-next-commands-options SUCCESS in 43m 13s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-arch-fedora SUCCESS in 44m 15s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-ubuntu SUCCESS in 13m 35s (non-voting)
✔️ system-test-fedora-coreos-stable-commands-options SUCCESS in 53m 33s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-arch-fedora SUCCESS in 44m 07s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-ubuntu SUCCESS in 10m 13s (non-voting)
✔️ system-test-fedora-coreos-testing-commands-options SUCCESS in 53m 41s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-arch-fedora SUCCESS in 44m 17s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-ubuntu SUCCESS in 13m 24s (non-voting)

@debarshiray debarshiray left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me! Just one question below.

Comment thread .zuul.yaml Outdated
Comment thread playbooks/setup-env.yaml
@debarshiray

Copy link
Copy Markdown
Member

It seems like we can land the first commit independently from the rest of the work and the CI turning green. It seems useful on its own, and it will keep things moving. Is it sufficiently well settled already?

I went ahead and merged the first commit through #1829

debarshiray pushed a commit to Rolv-Apneseth/toolbox that referenced this pull request Jul 30, 2026
containers#1806

Assisted-by: Claude (claude-opus-4-6)
Signed-off-by: Rolv Apneseth <rolv.apneseth@gmail.com>
@debarshiray

Copy link
Copy Markdown
Member

Rebased on main.

@debarshiray debarshiray left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops! I hit "submit review" too soon.

Could you please remove the Assisted-by: Claude (claude-opus-4-6) trailers from the Git commit messages?

I never wrote Assisted-by: Stack Overflow or Assisted-by: #gnome-hackers or similarly some API documentation, books, debuggers, other code bases, etc. in my Git commit messages in all my years of copy-pasting code from the Internet. Why care about copying any and every snippet of code written by artificial intelligence when we barely bate an eyelid over doing the same for human intelligence? :)

It seems to me that you are taking responsibility for the correctness of the code, and I am doing the same as the reviewer, so the generative AI is irrelevant for correctness. There's a big question mark about copyright, and that would be impossible to work out from a trailer in the Git commit message.

Beyond the aesthetic or philosophical nitpicking, in concrete terms, I don't want the Git commit messages of free software projects to turn into billboards offering no-cost advertising for big AI companies.

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor Author

Sure yes, I'll remove them. It was more about disclosure rather than wanting to give credit, but I understand your points.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/8c6db3e61c3f4ac898859c1145d28816

✔️ unit-test SUCCESS in 3m 13s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 30s
✔️ unit-test-restricted SUCCESS in 2m 17s
✔️ system-test-fedora-rawhide-commands-options SUCCESS in 46m 29s
✔️ system-test-fedora-rawhide-runtime-environment-arch-fedora SUCCESS in 28m 25s
✔️ system-test-fedora-rawhide-runtime-environment-ubuntu SUCCESS in 7m 57s
✔️ system-test-fedora-44-commands-options SUCCESS in 39m 46s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 28m 37s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 9m 22s
✔️ system-test-fedora-43-commands-options SUCCESS in 40m 23s
system-test-fedora-43-runtime-environment-arch-fedora NODE_FAILURE Node(set) request 200-0000157491 failed in 0s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 9m 26s
✔️ system-test-fedora-42-commands-options SUCCESS in 33m 53s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 30m 51s
system-test-fedora-42-runtime-environment-ubuntu NODE_FAILURE Node(set) request 200-0000157495 failed in 0s
system-test-fedora-coreos-next-commands-options NODE_FAILURE Node(set) request 200-0000157496 failed in 0s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-arch-fedora SUCCESS in 35m 51s (non-voting)
✔️ system-test-fedora-coreos-next-runtime-environment-ubuntu SUCCESS in 11m 45s (non-voting)
system-test-fedora-coreos-stable-commands-options NODE_FAILURE Node(set) request 200-0000157499 failed in 0s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-arch-fedora SUCCESS in 36m 19s (non-voting)
✔️ system-test-fedora-coreos-stable-runtime-environment-ubuntu SUCCESS in 12m 00s (non-voting)
✔️ system-test-fedora-coreos-testing-commands-options SUCCESS in 45m 04s (non-voting)
system-test-fedora-coreos-testing-runtime-environment-arch-fedora NODE_FAILURE Node(set) request 200-0000157503 failed in 0s (non-voting)
✔️ system-test-fedora-coreos-testing-runtime-environment-ubuntu SUCCESS in 11m 32s (non-voting)

Add system tests across the stable, next and testing Fedora CoreOS
streams, with some minor refactoring for shared dependency setup with
Fedora. Also added a cat /etc/os-release to the common dependency setup
as it may also be helpful on Fedora runs.

containers#714

Signed-off-by: Rolv Apneseth <rolv.apneseth@gmail.com>
containers#1806

Signed-off-by: Rolv Apneseth <rolv.apneseth@gmail.com>
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded.
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/local/buildset/f040c0cddd124c11817c674f6e44b5a4

✔️ unit-test SUCCESS in 2m 14s
✔️ unit-test-migration-path-for-coreos-toolbox SUCCESS in 3m 15s
✔️ unit-test-restricted SUCCESS in 2m 16s
✔️ system-test-fedora-rawhide-commands-options SUCCESS in 40m 50s
✔️ system-test-fedora-rawhide-runtime-environment-arch-fedora SUCCESS in 28m 50s
✔️ system-test-fedora-rawhide-runtime-environment-ubuntu SUCCESS in 7m 23s
✔️ system-test-fedora-44-commands-options SUCCESS in 47m 20s
✔️ system-test-fedora-44-runtime-environment-arch-fedora SUCCESS in 32m 56s
✔️ system-test-fedora-44-runtime-environment-ubuntu SUCCESS in 7m 42s
✔️ system-test-fedora-43-commands-options SUCCESS in 49m 46s
✔️ system-test-fedora-43-runtime-environment-arch-fedora SUCCESS in 36m 05s
✔️ system-test-fedora-43-runtime-environment-ubuntu SUCCESS in 8m 44s
✔️ system-test-fedora-42-commands-options SUCCESS in 32m 24s
✔️ system-test-fedora-42-runtime-environment-arch-fedora SUCCESS in 32m 28s
✔️ system-test-fedora-42-runtime-environment-ubuntu SUCCESS in 7m 53s
✔️ system-test-fedora-coreos-next-commands-options SUCCESS in 51m 57s
✔️ system-test-fedora-coreos-next-runtime-environment-arch-fedora SUCCESS in 34m 23s
✔️ system-test-fedora-coreos-next-runtime-environment-ubuntu SUCCESS in 11m 51s
✔️ system-test-fedora-coreos-stable-commands-options SUCCESS in 51m 27s
✔️ system-test-fedora-coreos-stable-runtime-environment-arch-fedora SUCCESS in 41m 40s
✔️ system-test-fedora-coreos-stable-runtime-environment-ubuntu SUCCESS in 11m 15s
✔️ system-test-fedora-coreos-testing-commands-options SUCCESS in 42m 21s
✔️ system-test-fedora-coreos-testing-runtime-environment-arch-fedora SUCCESS in 34m 06s
✔️ system-test-fedora-coreos-testing-runtime-environment-ubuntu SUCCESS in 8m 35s

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor Author
1..313
# test suite: Set up
# test suite: Tear down
not ok 1 setup_suite
# (from function `_setup_docker_registry' in file test/system/libs/helpers.bash, line 183,
#  from function `setup_suite' in test file test/system/setup_suite.bash, line 70)
#   `_setup_docker_registry' failed with status 126
...
# Adding password for user user
# Trying to pull quay.io/toolbox_tests/registry:latest...
# Getting image source signatures
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:cdee6680187b23f826eb651b125a7f16245aaa0381d7ed46a1b666e606c75d80
# Copying blob sha256:4980e9f75679a80a546f5d8c386ea8f77360100f2d8e8b0ae8782dab79f5d995
# Copying blob sha256:7a854618fb4b3af401a78181e3bf7ca1d9608493f822839f254ac0ec2a36a2df
# Copying blob sha256:792d5e381e06530261139997a960a2dd7cc771af1d6c8ddde680468c2082fcdd
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:0e99fe58b59645892107c30b0f25fed2e0777d6d94ce22101d1b8988d1ea06a5
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Writing manifest to image destination
# fea5a12cde107bb407bc44ede6dd9edea1d2b4171cd8e52b0cb330bf45e517e1
# Error: OCI runtime error: crun: unknown version specified
# Untagged: quay.io/toolbox_tests/registry:latest
# Deleted: fea5a12cde107bb407bc44ede6dd9edea1d2b4171cd8e52b0cb330bf45e517e1
# bats warning: Executed 1 instead of expected 313 tests

Not really sure what changed to cause this though. Any ideas @debarshiray?

@debarshiray

Copy link
Copy Markdown
Member
1..313
# test suite: Set up
# test suite: Tear down
not ok 1 setup_suite
# (from function `_setup_docker_registry' in file test/system/libs/helpers.bash, line 183,
#  from function `setup_suite' in test file test/system/setup_suite.bash, line 70)
#   `_setup_docker_registry' failed with status 126
...
# Adding password for user user
# Trying to pull quay.io/toolbox_tests/registry:latest...
# Getting image source signatures
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:cdee6680187b23f826eb651b125a7f16245aaa0381d7ed46a1b666e606c75d80
# Copying blob sha256:4980e9f75679a80a546f5d8c386ea8f77360100f2d8e8b0ae8782dab79f5d995
# Copying blob sha256:7a854618fb4b3af401a78181e3bf7ca1d9608493f822839f254ac0ec2a36a2df
# Copying blob sha256:792d5e381e06530261139997a960a2dd7cc771af1d6c8ddde680468c2082fcdd
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:0e99fe58b59645892107c30b0f25fed2e0777d6d94ce22101d1b8988d1ea06a5
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Writing manifest to image destination
# fea5a12cde107bb407bc44ede6dd9edea1d2b4171cd8e52b0cb330bf45e517e1
# Error: OCI runtime error: crun: unknown version specified
# Untagged: quay.io/toolbox_tests/registry:latest
# Deleted: fea5a12cde107bb407bc44ede6dd9edea1d2b4171cd8e52b0cb330bf45e517e1
# bats warning: Executed 1 instead of expected 313 tests

Not really sure what changed to cause this though. Any ideas @debarshiray?

Me neither - I don't know why the tests running on Ubuntu 22.04 hosts started failing.

I am guessing that it's something to do with GitHub and/or Ubuntu's infrastructure.

The failed job from this PR has:

0 upgraded, 45 newly installed, 0 to remove and 17 not upgraded.

... and the last successful job from #1829 has:

1 upgraded, 36 newly installed, 0 to remove and 83 not upgraded.

@debarshiray

Copy link
Copy Markdown
Member
1..313
# test suite: Set up
# test suite: Tear down
not ok 1 setup_suite
# (from function `_setup_docker_registry' in file test/system/libs/helpers.bash, line 183,
#  from function `setup_suite' in test file test/system/setup_suite.bash, line 70)
#   `_setup_docker_registry' failed with status 126
...
# Adding password for user user
# Trying to pull quay.io/toolbox_tests/registry:latest...
# Getting image source signatures
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:cdee6680187b23f826eb651b125a7f16245aaa0381d7ed46a1b666e606c75d80
# Copying blob sha256:4980e9f75679a80a546f5d8c386ea8f77360100f2d8e8b0ae8782dab79f5d995
# Copying blob sha256:7a854618fb4b3af401a78181e3bf7ca1d9608493f822839f254ac0ec2a36a2df
# Copying blob sha256:792d5e381e06530261139997a960a2dd7cc771af1d6c8ddde680468c2082fcdd
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:0e99fe58b59645892107c30b0f25fed2e0777d6d94ce22101d1b8988d1ea06a5
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
# Writing manifest to image destination
# fea5a12cde107bb407bc44ede6dd9edea1d2b4171cd8e52b0cb330bf45e517e1
# Error: OCI runtime error: crun: unknown version specified
# Untagged: quay.io/toolbox_tests/registry:latest
# Deleted: fea5a12cde107bb407bc44ede6dd9edea1d2b4171cd8e52b0cb330bf45e517e1
# bats warning: Executed 1 instead of expected 313 tests

Not really sure what changed to cause this though. Any ideas @debarshiray?

Me neither - I don't know why the tests running on Ubuntu 22.04 hosts started failing.

I am guessing that it's something to do with GitHub and/or Ubuntu's infrastructure.

Restarting the failed job brought it somewhat in line with the last successful one:

1 upgraded, 36 newly installed, 0 to remove and 83 not upgraded.

Let's hope it works out. :)

@Rolv-Apneseth

Copy link
Copy Markdown
Contributor Author

Me neither - I don't know why the tests running on Ubuntu 22.04 hosts started failing.
I am guessing that it's something to do with GitHub and/or Ubuntu's infrastructure.

Restarting the failed job brought it somewhat in line with the last successful one:

1 upgraded, 36 newly installed, 0 to remove and 83 not upgraded.

Let's hope it works out. :)

Well that's odd. Hopefully it's not a recurring flake then

@debarshiray

Copy link
Copy Markdown
Member

Me neither - I don't know why the tests running on Ubuntu 22.04 hosts started failing.
I am guessing that it's something to do with GitHub and/or Ubuntu's infrastructure.

Restarting the failed job brought it somewhat in line with the last successful one:

1 upgraded, 36 newly installed, 0 to remove and 83 not upgraded.

Let's hope it works out. :)

Well that's odd. Hopefully it's not a recurring flake then

Yeah. I have seen something like this once or twice before on rare occasions. I am relatively clueless about the internals of Ubuntu, so I restart the job and hope for the best. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Fedora CoreOS to continuous integration tests

2 participants