ci: make the compiler cache outlive the build container - #33
Open
hanthor wants to merge 29 commits into
Open
Conversation
The sccache client has been installed for every lane since b4f6313, and for WebKitGTK it cached nothing. Run 33582351064 shows why: the recipe wraps all 8955 of its compile actions in the client, but nothing inside the container sources /work/tools/sccache.env, so the credentials for the GitHub Actions backend never reach it. sccache falls back to /root/.cache/sccache -- a directory inside the container -- and that goes away when the container does. Both shards therefore compiled every object from scratch and threw the result away, twice per run. Two changes fix it. The recipe sources the environment file, as mozjs140 already does. And the environment file now points SCCACHE_DIR at a directory on the /work mount, which the lane restores from ghcr.io/<owner>/utah-packages-ccache:<package> before the build and publishes after it. The Actions cache backend is dropped: it is capped at 10 GB per repository and expires after a week, and two ports of WebKit with debuginfo would evict each other and everything else out of it. A GHCR repository has neither limit and costs nothing for a public repository. Only webkitgtk, webkit2gtk4.1 and mozjs140 opt in, through "compiler_cache" in the source manifest. Restoring and publishing several gigabytes costs a few minutes, which the 170 packages that build in under ten would pay for nothing. The cache is published even when the build fails, and read on pull requests: an sccache entry is addressed by the hash of the preprocessed source and the compiler flags, so a partial compile still leaves objects a later build can use, and no entry can be looked up by a build that would not have produced it. That same property is why the cache action is no longer hashed into a package build identity. It was: tuning the cache changed the key of all 178 packages and rebuilt the entire factory, which is the opposite of what a cache is for. SCCACHE_IDLE_TIMEOUT is now 0. The default ten minutes ended the server during the link and debuginfo phases, which is why the statistics at the end of a WebKitGTK build reported seven compile requests for a build that made 8955. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
The first version of this never published anything. Run 33616365993 proves the important half works -- mozjs140 reported Cache location Local disk: "/work/sccache" Compile requests 378, cache misses 298 so the objects now land on the /work mount instead of inside the container. The publish step then skipped them in silence: find: work/sccache/4: Permission denied find: work/sccache/preprocessor: Permission denied The build container runs as root and sccache creates its shard directories 0700, so the whole tree belongs to root and the runner user cannot read it. The content test was `test -n "$(find ...)"`, and find writes those errors to stderr and nothing to stdout, so the test failed, the step exited 0, and the job went green having cached nothing. Take ownership first. Two things around it. The build context for the cache image was the repository root, which sent the built RPMs and the seeded accumulator through podman build for no reason; it is now the cache directory alone. And the host-side statistics step is gone, along with the SCCACHE_DIR it needed: pointing a second client at the directory starts a server of its own and prints its own zeroes, which is precisely what hid the original fault for two runs. The recipes print the statistics from inside the container, where the compiling happens. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Member
Author
|
First run gave a partial result and a bug, both now addressed in The core fix works. mozjs140 in run 33616365993: Previously that read The publish step then dropped them silently. The build container runs as root and sccache creates its shard directories 0700, so the tree is unreadable to the runner user: My content guard was Two related changes in the same commit:
The wave already running on the previous commit keeps its RPMs and checkpoints; it just will not publish a cache. The next wave fills it. Generated by Claude Code |
The factory promises exactly config/upstream-sources.json: every entry there is a package it builds and publishes. Nothing checked that the published repository contains them, and two paths lead to one that does not. prepare treats a package as already published when it finds it in the :building accumulator, which is seeded from every wave including failed ones. publish seeds its candidate from :latest alone. A package that only ever reached the accumulator is therefore skipped by the build and absent from the candidate, and no later step looks for it. The gates that do exist cannot catch that. precedence ranks only what the current run built, so a package nobody built this time is outside its view. The Hummingbird-only consumer transaction enables the base OS repository beside the candidate -- it has to, because Hummingbird supplies part of the closure through the bootc base RPM database -- so a contract package the factory failed to provide resolves from Hummingbird instead and the transaction still succeeds. Both gates pass and the repository ships short. tools/repository_contract.py asserts the promise directly, in publish, before the metadata is signed and the image is pushed: every configured package appears in the candidate build manifest, and every RPM that manifest names is present with the checksum recorded when it was built. The last part matters as much as the first -- a half-merged artifact download leaves a file of the right name and the wrong contents, which existence alone would accept. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
rebuild3 / build (malcontent-bootstrap) in run 33624473308 died at the accumulator seed with Error: no command or entrypoint provided, and no CMD or ENTRYPOINT from image ##[error]Process completed with exit code 125. Every image this repository copies out of -- :building, :latest, and the compiler cache added in this branch -- is FROM scratch plus a COPY, so none declares a command, and podman 5 refuses to create a container from such an image unless the command is on the line. Six call sites had the same shape. It is not run: the container exists only so podman cp can read a path out of it, and it is removed immediately after. The step reads as tolerant -- it opens with `set -uo pipefail` and every other command ends in `|| true` -- but the job shell is `bash -e`, so this one unguarded assignment took the whole lane down. Sibling jobs in the same stage survived only because a pull that fails skips the block entirely. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Member
Author
|
Merge-when-green pass. Not merging: run 114 for the head commit is still queued, so this PR is not green yet. Two things found in the meantime. A pre-existing crash, now fixed here ( Every image this repository copies out of — What made it fatal rather than noisy: the step opens Cache status so far. The restore and publish steps behave correctly on packages that do not opt in — Generated by Claude Code |
Member
Author
|
The compiler cache works end to end. Evidence from run 33656040424 (head A 64% hit rate on the second run of an unchanged source, restored from GHCR and pushed back. Before this branch the same lane reported The podman fix cleared the failures it was meant to. Both WebKitGTK shards are compiling now with the cache steps wired, so the next report should have their hit rate and durations. Not merging yet: the wave is still in progress. Note that auto-merge cannot be enabled on this repository — the API returns "Auto-merge is not enabled for this repository" — so this is being merged by hand once every check is green. Generated by Claude Code |
Member
Author
|
WebKitGTK, the build this branch exists for, in run 33656040424:
Both succeeded. Against the history:
The lane's critical path is now under an hour, from five hours. 2817 of 2819 compile actions came out of the cache; the two misses are the only translation units whose preprocessed form changed. mozjs140 in the same run: restored 299M, 192 hits / 106 misses (64.43%), pushed 300M back. Lower because its cache was filled by a partial earlier run rather than a complete one; it should climb. The podman fix is holding — Still running: stages 2 to 4 and then Generated by Claude Code |
The publish gate caught this on run 33656040424: webkitgtk: webkitgtk-debugsource-2.53.91-1.hum1.bfin.x86_64.rpm does not match its recorded checksum Both shards build the same source package, and RPM names the debug packages after it, so each emitted webkitgtk-debuginfo and webkitgtk-debugsource at the same NEVR with different contents. The lanes' artifacts extract into one directory, so whichever landed second overwrote the first, and the losing shard's manifest no longer described the file on disk. The `-debugsource` mismatch was the visible half; `-debuginfo` collided the same way. `%global _debugsource_packages 0` was supposed to keep the GTK 3 shard from producing a debugsource package. It did not: find-debuginfo still ran with -S and the package was built. It also would not have helped, since it says nothing about -debuginfo. Give the GTK 3 shard its own source name instead, which gives it its own debug namespace and its own /usr/src/debug base. Nothing depends on that name: every binary package is declared with `%package -n`, there is no bare %files section so no package named after the source is ever produced, and Source0 and %autosetup spell the tarball out literally. It is webkitgtk4.1, not webkit2gtk4.1, which is already a subpackage. With neither bcond given, as on a plain rpmbuild, the name stays webkitgtk exactly as Fedora ships it. The gate reported this as a checksum mismatch on the shard that lost the race, which names the victim rather than the fault. It now checks for two packages recording the same filename with different contents and says so directly. Identical contents are not a collision -- whichever lands last is the same file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
The contract gate stopped run 33682968939 with ~185 of 187 packages reported as "the factory builds it, the candidate does not carry it". That is the gate working, and it names a real defect rather than a fault in the shards. `prepare` decides a package is already published by looking in the `:building` accumulator as well as `:latest`, so an incremental run rebuilds only what changed -- here, the two WebKitGTK shards. `publish` then seeded its candidate from `:latest` alone. Everything the accumulator had gathered across previous waves was in neither the seed nor this run's artifacts, so the candidate carried two packages. `:latest` predates the build manifest entirely, which is why nearly every entry came back missing rather than merely stale. Seed from both tags, lowest precedence first: `:latest` is the last contract-gated repository, `:building` accumulates every wave since and so is at least as new, and this run's artifacts land last and outrank both. Seeding from the accumulator is safe only because of what now follows it -- a half-finished accumulator fails the gate instead of publishing short, which is the whole point of having the gate. Two supporting changes. The malcontent bootstrap pass is deleted from the candidate before it ships: it breaks the flatpak-libs -> libmalcontent-0.so.0 cycle within one run and has no parental controls UI, so it must never reach a consumer. The delete ran after the gate, so the gate was inspecting a repository that differed from the published one. It now runs before, and the gate ignores outputs whose release is 0.bootstrap -- the factory does build that package, so it stays promised, but demanding files publish deliberately removes would fail every incremental run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
…name `prepare` reported "will build 23 of 187 packages" on every run, and it was the same 23 each time -- including both WebKitGTK shards, samba and mesa, which is where the wall clock went. They were not being rebuilt because anything had changed. The skip list is computed from `published`, harvested out of repodata, which lists BINARY package names. The name being looked up is the SOURCE package name from config/upstream-sources.json. For a source package that ships no binary of the same name the lookup can never succeed, so `name not in published` is permanently true and the package rebuilds forever. That is not an edge case: grub2 ships grub2-common and grub2-tools, mesa ships mesa-libGL and mesa-dri-drivers, wayland ships libwayland-client, pycairo ships python3-cairo, rust-just ships just, and webkitgtk ships webkitgtk6.0 and webkit2gtk4.1. Twelve of the thirteen entries checked by hand had no binary matching their source name; samba, which does, was selected for a different and legitimate reason. The identity check next to it does not have that flaw. It is keyed on the source name, and it is strictly stronger than comparing a name and a version: the build key covers the recipe, the source lock, factory policy and the Hummingbird base, and outputs_match then requires every RPM it recorded to be present with the checksum it was built with. So when a manifest entry exists it decides alone, and the repodata heuristic stays as the fallback for when there is none -- reading the published Pages repository rather than the accumulator. The consequence is only wasted time, never a short repository: a package wrongly skipped here is one the publish contract gate refuses to publish without. tests/test_prepare_selection.py extracts the decision block from the workflow and executes it rather than restating it, so the copy cannot drift. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
The publish gate now passes -- the candidate carries all 187 packages -- and
the Hummingbird-only consumer transaction fails immediately after it:
mesa-libGL-26.2.1-1.hum1.bfin.x86_64 from factory requires
mesa-dri-drivers(x86-64) = 26.2.1-1.hum1.bfin, but none of the providers
can be installed
nothing provides libSPIRV-Tools.so()(64bit)
nothing provides libsensors.so.4()(64bit)
nothing provides libxshmfence.so.1()(64bit)
mesa-libGL requires mesa-dri-drivers at the same NEVR, so an image that
installs the one installs the other, and mesa-dri-drivers links three
libraries nothing can supply. Hummingbird has 3,510 binary packages and none
of them is spirv-tools, lm_sensors, libxshmfence or any mesa: checked against
its primary metadata rather than assumed. The factory did not build them
either, so the closure was open on both sides.
This was not reached before because publish never got past the contract gate.
It is a long-standing gap in the repository's content, not a regression from
the shard or seeding work.
Import all three from Fedora rawhide with the existing closure tooling. Each
ships the exact soname the resolver asked for: spirv-tools installs an
unversioned libSPIRV-Tools.so, libxshmfence installs libxshmfence.so.1, and
lm_sensors carries Fedora's soname-revert patch, which takes LIBMAINVER from
5 back to 4 and so installs libsensors.so.4. They are leaf libraries, so they
need no stage override.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Hummingbird is upstream-first. The factory was not: config/upstream-sources.json froze whatever version was imported, and nothing moved it. The machinery to do better was already here and simply never wired up -- tools/dist_git.py is a Koji-gated dist-git checker whose docstring says "the workflow turns them into reviewable update branches", and no workflow called it. Meanwhile detect-rawhide-updates.yml announced "Fedora is a compatibility build root, not a source-update feed", which is true of the build root and wrong as a policy for sources. tools/track_upstream.py compares every entry against Fedora dist-git rawhide and reports two different kinds of drift: Version drift, where our entry is older than rawhide. Following upstream means taking rawhide's version, but only once Koji reports a COMPLETE build of that exact NVR. dist-git advancing is not proof Fedora built it, and promoting on a commit alone can name a version no binary exists for. Location drift, where the entry fetches from the upstream project rather than Fedora's lookaside. This is the one that actually bites. wireguard-tools is pinned to git.zx2c4.com/wireguard-tools/snapshot/, which generates tarballs per request and 404s -- while rawhide ships that exact version with a byte-identical sha512, reachable from the lookaside. Verified: the proposed URL returns 200 and the download hashes to the digest already recorded. 116 of 190 entries already use the lookaside; 74 point at hosts that rewrite and expire. The sha512 stays in both cases. It is not a version pin -- it is what makes the pipeline fail closed when a tarball is re-rolled underneath us, which is exactly the risk that following upstream raises rather than lowers. track-upstream.yml runs this daily and opens a pull request. It proposes, never applies: editing the source manifest invalidates every package's build identity and rebuilds the whole factory, so these land as their own reviewed change rather than arriving in the middle of a wave. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Run 33714581543 built 202 of 205 jobs green -- including the three new mesa providers and both WebKitGTK shards -- and one package failed, which was enough to block publish: all source URLs failed: https://git.zx2c4.com/wireguard-tools/snapshot/wireguard-tools-1.0.20260223.tar.xz: failed to fetch after 3 attempts: HTTP Error 404: Not found cgit generates snapshot tarballs per request and stops serving them when the tag moves, so that URL was always going to expire. Fedora rawhide still ships this exact version, and its lookaside holds a byte-identical artifact: the sha512 recorded here already matches Fedora's sources file, and the download hashes to it. Only the location changes -- not the version, not the bytes, not the digest we verify against. Produced by tools/track_upstream.py, which classified it as location drift rather than version drift precisely because rawhide is on the same version. Also stop --apply from sorting keys when it writes the manifest. json.load preserves insertion order, so dumping without sort_keys leaves every untouched entry byte-identical; sorting rewrote all 190 entries and buried a one-line source change in a 632-line diff. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
The mesa providers worked. Run 33729693293 reached the Hummingbird-only consumer transaction with no mention of libSPIRV-Tools.so, libsensors.so.4 or libxshmfence.so.1 -- a949f55 resolved those completely. The gate reports only the first unsatisfiable set, so closing one layer exposes the next, and it named 29 problems that reduce to three missing providers: libstemmer.so.0, needed by libtinysparql. This is the one that hurt: nearly every problem in the list is a cascade from it. Hummingbird ships libtinysparql but not the libstemmer it links, so its own package is unsatisfiable, and gtk4 requires libtinysparql-3.0.so.0 -- which takes down libadwaita, zenity, gtk3, gnome-session, gnome-desktop4, xdg-desktop-portal-gnome and openssh-askpass with it. Fedora's snowball builds a libstemmer subpackage. /usr/bin/xprop, needed by at-spi2-core. Fedora's xprop ships %{_bindir}/xprop. xdg-user-dirs, needed by xdg-user-dirs-gtk, which gnome-shell requires and gsconnect requires after that. Fedora's xdg-user-dirs is the package itself. Each was verified against rawhide dist-git before importing rather than guessed: snowball declares %package -n libstemmer, xprop lists the binary in its %files, and xdg-user-dirs is a direct name match. The factory promises 193 packages now. That the gate keeps finding these is the point of having it. Every one is a package Utah would have installed into a broken closure. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Layer two worked: libstemmer, xprop and xdg-user-dirs are gone from the gate's
output. Rather than keep paying a three-hour rebuild per layer, this reads the
whole failure instead of its tail. dnf reports every problem it found, so the
complete set was already there: 29 problems naming 90 distinct unsatisfiable
capabilities, which reduce to 44 source packages.
The pattern is not a tail of stragglers. Hummingbird is a server base -- 3,510
packages, no desktop stack -- and the factory's desktop packages link a long
list of media and desktop libraries that neither side ships:
gstreamer1-plugins-base alone needs opus, orc, libXv, cdparanoia, libogg,
libtheora, libvisual and libvorbis, and it is what gtk4 requires, so that
one package gates most of the GNOME stack.
webkitgtk6.0 needs jpegxl, flite, enchant2, hyphen, libmanette and
xdg-dbus-proxy; pipewire-libs needs sbc, libldac, liblc3, libsndfile,
spandsp, libebur128, fdk-aac-free and bluez; fwupd needs libxmlb and passim;
colord-libs needs libgusb, which is what actually took gtk3 down.
Some are Hummingbird's own packages failing against Hummingbird: its lvm2
requires libnvme and device-mapper-persistent-data, which it does not carry.
Supplying them from the factory is the only lever available here.
Every name was checked against rawhide dist-git before importing, which caught
three that a guess would have got wrong: libjxl is built by jpegxl, ldacBT by
libldac, and libldac's spec sets Version from %{sonamebase} so the importer
recorded a macro rather than 2.0.2.6. Four more -- cdparanoia, hyphen,
libasyncns and libtheora -- record MD5 in dist-git rather than SHA-512, so the
importer skipped them; each was fetched from the md5 lookaside, its digest
confirmed, and a sha512 computed from the bytes actually downloaded. The
integrity check stays exact for them.
Two are worth a second opinion rather than being slipped in. ffmpeg is here
because pipewire-libs-extra links libavcodec.so.62, and Utah deliberately does
not enable fedora-multimedia, so building codecs in the factory is a policy
choice. anaconda-webui is here because anaconda-live requires it, which is ISO
tooling rather than the desktop runtime. Both are needed for the transaction to
resolve; neither is obviously wanted forever.
237 packages promised now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Run 124 built 229 of 237 and failed eight. Reading all three pages of the check runs rather than the first -- the API returns 100 per page and reports the rest only in total_count -- gives the whole list, and five of the eight reduce to two causes. libbluray is the larger one. rawhide dist-git has moved to 1.5.0, which is libbluray.so.4, while the composed repository the buildroot resolves against still ships 1.4.0 and everything in it, Fedora's own libavformat-free included, links so.3. Building 1.5.0 excludes Fedora's 1.4.0 by name -- that is how the factory keeps Fedora from answering for what it rebuilds -- and then nothing provides so.3, so libavformat-free became uninstallable and took localsearch, pipewire-libs-extra and, through gvfs, nautilus-python with it. The factory cannot rebuild Fedora's compose ahead of Fedora. It can decline the feature: gvfs gets -Dbluray=false unconditionally where Fedora sets it only on RHEL, ffmpeg gets a libbluray bcond defaulting off, and the entry comes out of the manifest. Blu-ray disc navigation is not something a Utah machine has a drive for, and declining it is what keeps one soname in play. When rawhide's compose carries 1.5.0, the tracker will propose it back. ffmpeg's own failure was pkgconfig(vapoursynth) >= 79, which Fedora has and Hummingbird does not. vapoursynth is a frameserver rather than a codec and nothing in the contract links it, so it gets a bcond too rather than dragging zimg and a Python extension in to satisfy a BuildRequires whose output no consumer asks for. ffmpeg matters here: pipewire-libs-extra links libavcodec.so.62 at runtime and Hummingbird ships no libav at all. The other three are each their own: protobuf pulled Fedora maven-lib into a buildroot that already had Hummingbird's maven3.9, and the two disagree about twenty files under /usr/share/maven. Nothing in the contract asks for protobuf-java, so the entry builds --without java rather than arbitrating between two Java stacks. python-typing-inspection ran %pyproject_patch_dependency in %prep, which imports packaging, before the generated BuildRequires are resolved. It now asks for python3-packaging by name. libvorbis lost its %changelog header in the import, so rpmautospec's expansion landed in the last %files section and every word of "* Thu Sep 03 2026 John Doe <packager@example.com>" was read as a path. libayatana-ido's %check calls xwfb-run, and xwayland-run does not depend on a compositor, so it died with "No such file or directory: 'weston'" before a test ran. Fedora's builders have weston for other reasons; this asks for it rather than dropping the suite -- the same call the system bus and USER in the lane script already make. 236 packages promised now. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
The closure commit flagged ffmpeg and anaconda-webui as worth a second opinion rather than being slipped in. Both are in: they are part of the vanilla Fedora stack, which is the bar, whether or not Utah installs with Anaconda. Since the recipes are Fedora's, this builds ffmpeg-free and libavcodec-free -- Fedora's codec policy, not RPMFusion's. The same page now records the two declines the last commit made, so the next reader finds the reasoning instead of an unexplained gap: libbluray, where rawhide dist-git is a soname ahead of the compose the build root resolves against, and vapoursynth, which is a frameserver nothing in the contract links. Both share one rule worth stating once -- where the factory and the Fedora compose disagree about a soname, the factory declines the feature rather than getting ahead of the compose it builds against, because adding the newer library is the move that strands Fedora's own packages. Documentation only. Build identity covers config/, the recipe directory, hummingbird.repo, the runtime contract and rebuild-lane.yml, so nothing here invalidates a build key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
…mised
Removing libbluray from the manifest did not take effect, and made things
worse. The lane seeds /work/prior from the `:building` accumulator -- shared
across every run and ref -- and then derives the Fedora exclusion list from
whatever names it finds there: what the factory builds, Fedora must not answer
for. That is right while an entry exists and wrong the moment one is removed.
The libbluray RPMs from the run before were still in the accumulator, so
Fedora libbluray stayed excluded with nothing left to provide the soname. The
factory libavformat-free needs it, so it became uninstallable, and with it
libheif, gdk-pixbuf2, glycin, graphviz and doxygen -- which is how ten stage 0
packages that have nothing to do with Blu-ray failed at once: wayland, zenity,
pulseaudio, mozc, libvisual, spandsp, sdl2-compat, libtheora,
xdg-desktop-portal and xdg-user-dirs-gtk.
So the accumulator is now filtered against what the manifest currently
promises before it becomes a repository. tools/promised_sources.py lists the
source names, and the lane drops any RPM whose %{SOURCERPM} is not among them,
beside the ICU-77 purge that already does this shape for stale ABI. Matching
on the source package rather than the binary is the same reasoning as that
one: a source build ships subpackages that require each other by exact NEVR,
so dropping one and keeping its siblings is worse than dropping none.
Four spellings can each be the real source name -- the entry, the recipe
directory, the dist-git name, and every Name: the spec declares, since
webkitgtk sets it per GTK port -- and all four are allowed rather than guessed
between. Five recipes set Name: from a macro this cannot expand, or from a
font template that never spells it; each resolves to its own recipe directory,
which is already in the list. The tests pin that, and pin the sed that strips
%{SOURCERPM} back to a name, which has to take the version and release off the
end because a source name may itself contain dashes and dots.
protobuf got past the maven conflict with --without java and failed later, in
%build:
Target "test_plugin" links to: absl::scoped_mock_log
but the target was not found
for four test targets. This factory builds abseil-cpp with
ABSL_BUILD_TEST_HELPERS OFF where Fedora builds it ON, because that graph is
LTO-heavy and OOMs the hermetic runner, and Fedora abseil-cpp is excluded, so
protobuf resolves against ours and cannot configure a suite %bcond_with check
never runs. It now sets protobuf_BUILD_TESTS OFF. The deviation is recorded at
both ends: the next package to want a test-only Abseil target will fail the
same way.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
The accumulator purge worked -- the exclusion list went from 1573 names to 1527 and libbluray and anaconda are gone from it -- and stage 0 failed anyway, worse than before. The purge was the right fix for the wrong layer. One layer up sits the same shape, and this time the factory caused it: libheif-1.23.1 from stages requires libavutil.so.60, no provider libavutil-free-8.1.2-4.fc44 from updates is filtered out by exclude filtering A stage resolves the factory packages from the accumulator, which is the run before; only later stages see what this run has built. So promising ffmpeg excludes the Fedora libav by name from every buildroot the moment the entry exists, while the factory replacement is a run behind -- and ffmpeg is itself stage 0. The accumulator libheif was linked against the Fedora 8.x libavutil.so.60, the factory ffmpeg 9.x provides a later one, and libheif became uninstallable. gdk-pixbuf2 needs glycin needs libheif, graphviz needs gdk-pixbuf2, doxygen needs graphviz, so twenty-odd stage 0 packages with no connection to video went down: opus, pango, wayland, pulseaudio, libthai, libtalloc, tecla, passim, openrgb, jpegxl and the rest. The invariant that was missing: nothing in stage 0 may link libav, because ffmpeg is in stage 0. Two ways to satisfy it, one each here. waypipe moves to stage 1. Nothing in the factory BuildRequires it, so it can simply build after ffmpeg and keep its video feature intact. libheif cannot move -- glycin is stage 0 and needs it -- so it declines the feature instead, the same call gvfs and ffmpeg already make for libbluray. A narrow %bcond ffmpeg 0 gates the ffmpegdec plugin only; %bcond bootstrap would have gone further and dropped rav1e, SvtEnc and sdl2 with it. The aom and dav1d decoder plugins remain, and those are what decode AVIF and HEIC here. Neither helps this run on its own, because stage 0 reads the accumulator, not this run: the stale libheif would still be there. So the purge beside the ICU-77 one now also drops every libav-linked RPM and every ffmpeg subpackage from the accumulator. That takes those names out of the exclusion list too, so Fedora answers for libav again until this run rebuilds ours. It stands in for a general rule worth stating: a newly promised source cannot be excluded from Fedora before the factory has built it once. tests/test_stage_zero_libav.py fails when a third package takes neither way out. It tracks %if nesting rather than grepping, so a gated BuildRequires passes and a bare one does not, and it pins ffmpeg being stage 0 as the premise the whole rule rests on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
gtksourceview4 build-requires pkgconfig(gladeui-2.0) for its Glade catalog, and the Fedora glade-devel drags in the Fedora webkit2gtk4.1, which needs libicuuc.so.77. The lane excludes the older Hummingbird libicu-77 globally so that only 78 is selectable -- a build that picked 77 produced RPMs the Hummingbird-only runtime could not install -- so that webkit cannot be installed and the transaction has nowhere to go. The lane comment on that exclusion already named this case: "Fedora build-only consumers that want 77 are rare and not part of the runtime." gtksourceview4 is one. What it loses is an XML file describing widgets to the Glade designer; nothing in Utah's runtime reads it, and Fedora itself drops it on RHEL 10 through the same bcond this flips. Deliberately a recipe edit rather than rpm_defines in the manifest. Build identity hashes the whole source manifest, so any manifest edit invalidates all 236 build keys and costs a full three-hour rebuild; a recipe edit invalidates only this one package, and everything else is served from the accumulator by identity match. Same reason the two remaining fixes should stay out of the manifest if they can. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
The run reached stage 4. nautilus-python, localsearch and pipewire-libs-extra -- the three that had failed every previous attempt -- are green, along with both webkit shards, samba, gvfs, mutter, gjs, gnome-control-center, malcontent and every checkpoint. Two new failures, evolution-data-server and gnome-shell, share one cause. Their buildroots resolve the Fedora webkit2gtk4.1-devel and webkitgtk6.0-devel, whose javascriptcore needs libicuuc.so.77, and Hummingbird libicu-77 cannot be installed beside the libicu-78 that our own libphonenumber, libical and mozjs140 require. Deadlock. Fedora webkit is only in the running because ours is not: the exclusion list for that lane has 1485 names and contains no webkit, no javascriptcore and no ffmpeg, while libheif, glycin, localsearch and pipewire-libs are all there. So webkit2gtk4.1 was absent from /work/prior even though both shards had built an hour earlier, and Fedora was left to answer for it. Not deleted, though -- both purges logged zero deletions in that lane and createrepo_c walked 1495 packages -- so this is an absence, not a removal, and I have not established which step drops them. What is established is that the libav purge is the one mechanism here that can delete a legitimate artifact, and that it is no longer needed: it existed because a newly promised source cannot be excluded from Fedora before the factory has built it once, and ffmpeg has now been built and accumulated. Its stale libheif is gone too, both because libheif no longer links libav and because accumulate has since run with a coherent set. So it comes out rather than being narrowed. The unpromised-source purge stays. That one is general, it is the fix for removing an entry from the manifest, and it is not implicated. If evolution-data-server still cannot see our webkit after this, the fault is in artifact propagation between lanes rather than in either purge, and that is where to look next. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
gtksourceview4 is green, so that fix worked. ffmpeg now fails instead, and it had built cleanly in the run before -- which corrects the hypothesis in ab315f0. The libav purge was not incidental to webkit; it was accidentally holding ffmpeg up, and removing it exposed what was underneath: libchromaprint-1.6.0-4.fc44 from fedora requires libavutil.so.60 libavutil-free-8.1.2-4.fc44 from updates is filtered out by exclude filtering ffmpeg build-requires pkgconfig(libchromaprint), and Fedora libchromaprint links the Fedora libavutil.so.60. Promising ffmpeg excludes the Fedora libav by name, so libchromaprint-devel cannot be installed and ffmpeg cannot configure. The purge had been deleting our own libav RPMs from the accumulator, which left the Fedora libav selectable and quietly kept this resolvable. The ffmpeg spec names this cycle already: ffmpeg (libavcodec-free) → chromaprint → ffmpeg, broken in Fedora by one bootstrap build and a rebuild of chromaprint against the result. The factory cannot do that, because it does not build chromaprint, so the cycle is permanent here rather than a bootstrap phase. It is declined instead: chromaprint is AcoustID audio fingerprinting and nothing in the runtime contract links it. The same call as vapoursynth and libbluray, and the third instance of the rule those two established -- where the factory and the Fedora compose disagree about a soname, decline the feature rather than get ahead of the compose. Only the recipe changes, so only ffmpeg is rebuilt; everything else is served from the accumulator by identity match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
ffmpeg is green, so ac8bf79 worked, and gstreamer1-plugins-bad-free fails on exactly the same block -- libchromaprint-devel wanting libavutil.so.60 while the Fedora libav is excluded. chromaprint is not an ffmpeg-specific problem: it is a property of promising ffmpeg at all, so no package here can build-require anything that links the Fedora libav. Only these two recipes mention chromaprint, so this closes the set. A narrow %bcond rather than turning off %{extras}, which gates assrender, avtp, bs2b, curl, dca, modplug and a dozen more. The switch moves out of the extras block so it fires on either condition, and the plugin leaves %files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Both of these are what Fedora does, which is also what keeps Hummingbird upstream-first: the divergences here were the factory improvising, not Hummingbird asserting anything. mozc stopped with Argument 'protobuf-devel < 4' matches only excluded packages and only once protobuf started building successfully -- while it failed, the Fedora protobuf was never excluded and mozc quietly used the 3.19.6 in the Fedora 44 compose. mozc genuinely wants the 3.x API, and mozc and ibus-mozc are both promised in bluefin-packages.toml, so dropping either it or protobuf would have been the factory inventing a policy. Fedora already solved this: rawhide carries a separate protobuf3 source package at 3.19.6 whose -devel subpackage Provides protobuf-devel = 3.19.6, satisfying the constraint beside the 33.5 that everything else links. So that package is imported, from rawhide dist-git with its provenance recorded, at the version rawhide ships. Its googletest Source1 needs no special handling -- source_pipeline already fetches bundled sources from the lookaside using the dist-git sources file. mozc moves to stage 1 so it builds after protobuf3 rather than racing it in stage 0, the same move waypipe made behind ffmpeg. Nothing build-requires mozc, so the stage is free. mozjs140 stopped with ERROR: Don t know how to translate x86_64-redhat-linux-gnu for rustc against rust-1.98.0-1.hum1. Fedora builds rustc with an extra x86_64-redhat-linux-gnu target and Mozilla configure looks the host triple up in rustc --print target-list; Hummingbird builds upstream rust, which knows only x86_64-unknown-linux-gnu. So the Rust toolchain now comes from Fedora, by the same global exclusion that already keeps libicu 77 out. That is the documented division of labour rather than an exception to it -- Fedora 44 is the other half of the buildroot, the way a compiler is, and upstream-first is a statement about what ships at runtime, where no Rust toolchain appears at all. Worth watching: this changes the compiler for every Rust package here, not just mozjs140 -- rust-just, waypipe, librsvg2, glycin, python-pydantic-core and firefox all built against the Hummingbird rust before. Fedora builds them all against its own, so this should hold, but if one of them regresses that is where to look. The patterns are version-anchored so rust-1.* cannot swallow rust-std-static or a rust-<crate>. libgudev is still open and is not addressed here. Its spec is byte-identical to rawhide, which carries no patches, so Fedora runs this same %check in mock and passes; the difference is the container, and I have not established which part of it. It has now failed twice, so it is real rather than a flake. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
077600d worked for everything except the package it was written for. mozc built against the newly imported protobuf3, gstreamer1-plugins-bad-free built with chromaprint declined, and the Rust packages that were the blast-radius worry -- waypipe, python-pydantic and the rest -- are all green. mozjs140 still failed, identically: ERROR: Don t know how to translate x86_64-redhat-linux-gnu for rustc [ 2/170] rust-0:1.98.0-1.hum1.x86_64 So --exclude=rust-1.*hum1 does not match. The Hummingbird rust was installed with that pattern on the command line, which is the whole answer: dnf did not read it as excluding rust-1.98.0-1.hum1. Exclude by name from the Hummingbird repository instead. That form is already proven in this lane -- it is what produces "filtered out by exclude filtering" for the Fedora side throughout the resolver output -- so it is the one to trust. Name-scoping is safe here because Fedora is the only other source of a Rust toolchain, which is exactly what this wants selected. libicu cannot use this form, since 77 and 78 share one name and only the older build is unwanted; its glob stays as it is. Worth noting for whoever reads the libicu line next: the same doubt now applies to it. libicu-77.1-2.1.hum1 has been turning up in transactions that had --exclude=libicu-77.*hum1 set, so that exclusion may never have worked either. It is left alone here because changing it needs its own evidence, and because a name-scoped exclusion would take libicu 78 with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
This is a judgement call the user delegated, on the basis of what Fedora does
and what Hummingbird is for. Recording the reasoning because the conclusion is
weaker than the others in this branch.
My earlier read was wrong and is corrected here. test-gudevdevice does not
need a live udev database. It builds a device entirely out of environment
variables:
g_setenv ("DEVPATH", "/devices/dev1", TRUE);
g_setenv ("TAGS", "tag1:tag2:tag3", TRUE);
g_setenv ("UDEV_DATABASE_VERSION", "1", TRUE);
udev_device = udev_device_new_from_environment (udev);
and the log shows that call returning NULL with EINVAL:
# MESSAGE: error is: 22, Invalid argument
# MESSAGE: tags list entry form udev device (nil): (nil)
so no container property explains it. What the test pins is libudev's
willingness to accept a hand-built environment, not anything libgudev does --
_g_udev_device_new simply propagates the NULL. libgudev's own behaviour stays
covered by the other three tests, which drive real devices through umockdev
and pass, and by the library's actual consumers here, all of which build.
What Fedora does: rawhide pairs libgudev 238 with systemd 262~rc1, carries no
patches, and does not hit this. The build root here resolves Hummingbird's
systemd 261.2 instead, which is where the behaviour differs. Fedora's habit
when a rebuild trips a test like this is a downstream patch with the reasoning
attached, which is what this is.
What Hummingbird is for: it ships upstream systemd ahead of the compose, and
the factory builds for that platform. A test whose fixture assumes a specific
udev database version is exactly the kind of thing that breaks when a platform
runs ahead. Holding all 237 packages -- and Utah behind them -- on one
synthetic fixture would invert the point of the exercise.
What this is not: the difference between systemd 261.2 and 262~rc1 has NOT
been root-caused, so this is carried as parity debt in
docs/targeting-hummingbird.md rather than presented as understood. It is the
one place in the 237 where the factory ships a package without running its
upstream suite in full, and it should be revisited when either systemd moves.
A test hardcoding UDEV_DATABASE_VERSION 1 is asserting an internal contract
libudev is free to change, which looks worth reporting upstream.
Recipe-only, so this rebuilds libgudev alone.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Root cause, reproduced rather than inferred. Mozilla's configure does not hand the autoconf triple to rustc; it runs "rustc --print target-list", groups the results by (cpu, endianness, os) and narrows that group to one rust target. For x86_64-redhat-linux-gnu the deciding narrowing was the raw OS, "linux-gnu", which matched x86_64-unknown-linux-gnu alone. rust 1.98 added the OpenEmbedded targets. x86_64-oe-linux-gnu shares the raw OS and the raw CPU with the generic target, so both narrowings now leave two, the last resort tries the vendor, "redhat" names no rust target, and configure stops: ERROR: Don't know how to translate x86_64-redhat-linux-gnu for rustc Replayed the selection against real target lists: on rust 1.94 the stock code returns x86_64-unknown-linux-gnu, on 1.98 it returns nothing, and the patch returns x86_64-unknown-linux-gnu on both. aarch64-redhat-linux-gnu breaks and recovers the same way; ppc64le and s390x were never ambiguous and are unchanged. Upstream has taken the same correlation in the same place -- the comment on mozilla-firefox main names "the *-oe-linux-gnu targets added in rust 1.98" -- but only maps the "pc" vendor config.guess emits onto rust's generic "unknown", which does not cover a distribution triple. This adds the fallback: a vendor that matches no candidate wants the generic target. Rawhide carries no such patch because rawhide has not rebuilt mozjs140 since that rust landed, which is the kind of latent break a rebuild-everything factory meets first. This also corrects 077600d and 6b83cd5. Both were written on the theory that Hummingbird's rust lacked a target Fedora's rust had. 6b83cd5 did make the buildroot take Fedora's toolchain -- rust-0:1.98.0-1.fc44 and rust-std-static from updates, confirmed in the run 133 transaction -- and mozjs140 failed byte-identically, because the vendor of rust was never the variable. Its version was. Neither commit is reverted here: they touch the lane workflow, so undoing them invalidates all 237 build identities and costs a full rebuild, and taking the Rust toolchain from Fedora is what Fedora does anyway. Every Rust package in the tree built green against it. Recipe-only, so this rebuilds mozjs140 alone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
The 237 all built, the contract gate passed -- "candidate repository carries
all 237 packages the factory promises" -- and publish then failed on the gate
after it, the one that installs the runtime contract from Hummingbird and the
factory alone. That gate had never run before: publish was skipped on every
previous run because something was red. So this is the first honest reading of
what the promised set actually needs, and it is short by a dependency closure.
Every problem in the transaction reduces to a capability neither Hummingbird
nor the factory provides. Twenty-six problems, eighteen missing sources:
libICE, libSM qt5-qtbase-gui, via mozc
xcb-util-{wm,image,keysyms,renderutil} same
mesa-demos glx-utils, same
libdatrie libthai, which we build, for pango
libnice gstreamer1-plugins-bad-free-libs, so gtk4,
so libadwaita, zenity, gnome-desktop4,
xdg-desktop-portal-gnome, gnome-tweaks
glib-networking libsoup3, so gnome-shell,
gnome-control-center, gvfs, geoclue2
gweather-locations libgweather, so gnome-settings-daemon
wireplumber pipewire-session-manager, so pipewire,
so mutter and pipewire-libs-extra
rtkit pipewire
adwaita-icon-theme-legacy adwaita-icon-theme, so gtk3
libdecor SDL3, since we build wayland
iputils dracut-network, so dracut-live
cockpit, python-bugzilla anaconda-webui
Imported from rawhide dist-git with provenance recorded, at the versions
rawhide ships, hash-addressed to the lookaside. All stage 0, which is where
203 of the existing 237 sit -- the stage 0 rule this tree enforces is about
soname skew across a run boundary (tests/test_stage_zero_libav.py), not about
depending on a sibling at all, and none of these link libav.
One gap is NOT closed here and needs a decision: anaconda-webui's browser.
Its rich dependency resolves to firefox on anything without one of the spin
identity packages, and neither Hummingbird nor the factory has firefox. It is
raised separately rather than guessed at, because building firefox on a
GitHub-hosted runner is a different order of undertaking from these eighteen.
Also fixes a real bug this exposed in the importer. It recorded the first
entry of the dist-git sources file as the upstream source, but that file is in
dist-git order and carries bundled archives too: iputils lists a bundled
ifenslave.tar.gz ahead of its own tarball. Source0 is now resolved from the
spec. The same bug had already landed on snowball, whose manifest entry
pointed at the snowball-data test corpus rather than snowball itself; it built
anyway because the pipeline fetches every archive in the sources file, but the
entry described the wrong thing. Corrected, with tests pinning the selection
and asserting the property across every multi-source recipe in the tree.
This is a manifest change, so it rebuilds all 255.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Closing the one gap the previous commit left open. anaconda-webui carries a
rich dependency that reads
(slitherer if (fedora-release-identity-budgie or ... or
fedora-release-identity-i3) else firefox)
so on anything not carrying a spin identity package -- which is the Hummingbird
bootc image -- it resolves to firefox, and neither Hummingbird nor the factory
has one. The user chose firefox over the alternatives, which were shipping a
spin identity package to select slitherer instead, or dropping anaconda-live
from the runtime contract.
Imported from rawhide dist-git with provenance recorded, at 155.0, and placed
in the stage1_heavy lane beside the two WebKit shards, which is the lane that
exists for compiles that want a runner to themselves. Opted into the compiler
cache and taught to use it: mach drives the C, C++, rust and host compiles
itself, so the cache goes in through the --with-ccache option the build system
provides rather than by wrapping CC, which would miss most of them.
One deliberate divergence from rawhide, using a switch the recipe already
provides: release_build is 0, so the build is Fedora's own non-PGO
configuration and carries the .npgo release tag the recipe appends for it. A
PGO build is two compiles with an instrumented run between them and needs a
wayland compositor for that run; Koji gives it a builder for most of a day,
this factory has four cores and a six-hour job limit. It also turns off the
WASI SDK sub-build, a cmake and cargo toolchain compile that runs before
Firefox itself starts. The cost is real and recorded in the spec: no
profile-guided optimisation, and no RLBox sandboxing of the media libraries,
so the browser is slower than Fedora's and one defence-in-depth layer thinner.
Worth revisiting if the factory ever gets a builder that can afford the full
build.
Two things the importer needed for this. Source0 is spelled with
%{?pre_version}, a conditional macro nothing defines on a release, so
undefined conditionals now expand to nothing -- without that the entry
recorded mochitest-python.tar.gz, the first line of the sources file, as the
upstream source of Firefox. And a new test asserts that every recipe opted
into the compiler cache actually reads the cache environment, because the flag
only mounts the client: opting a package in and leaving its %build alone gives
a silently uncached build, which on a compile this size is the difference
between fitting the runner and not.
Cancelled run 136 rather than let it finish. Its manifest predates firefox, so
none of its build identities survive this commit and none of its output would
have been reused; running it out would only have delayed the first attempt at
the long pole by two hours.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
cockpit was the only one of the nineteen new recipes that failed, and it did not fail on anything cockpit does. Its own test-auth stops at GLib-FATAL-WARNING: g_hmac_new: GLib HMAC is disabled for FIPS compliance which is a property of the platform, established rather than inferred: unpacking glib2-2.89.3-1.hum1 from Hummingbird's own repository and reading libglib-2.0.so.0.8903.0 shows that string present, "gnutls" absent from the binary entirely, and no "fips_enabled" anywhere. The disablement is compiled in and unconditional -- it applies on every Hummingbird system whether or not FIPS is enabled, not only in a build container. Fedora carries gnutls-hmac.patch and routes GHmac through GnuTLS so it keeps working; Hummingbird builds without that backend and the fallback is a stub that warns and returns NULL. cockpit-ws authenticates with GHmac, so a cockpit built here would install and fail to authenticate. Patching out the test would have shipped precisely that, which is the thing the publish gates exist to prevent. So anaconda-live joins slitherer in [unavailable]. That list already carried the reasoning -- "Utah uses the bootc-installer live path, not Anaconda WebUI or Slitherer" -- and this is the same path one level up: anaconda-live pulls anaconda-webui, which pulls cockpit-bridge, cockpit-ws, cockpit-networkmanager and cockpit-storaged. cockpit, python-bugzilla and firefox came in only to serve that chain and come back out with it; firefox leaves the heavy lane as well. The factory still builds anaconda-webui and anaconda-live as vanilla Fedora packages, which was the earlier call and is unchanged. The exception says only what Utah's runtime installs. The finding itself is not a decision about Hummingbird and should not be mistaken for one. Every GHmac consumer on that platform is affected, not just cockpit -- libsoup digest authentication, evolution-data-server, gvfs and gnome-online-accounts all reach for it -- so it wants reporting upstream with the binary evidence, which is written up in docs/targeting-hummingbird.md. Revisit here when Hummingbird's glib2 gains a working backend. Run 137 cancelled: it was building firefox for a contract entry that no longer exists, and cockpit had already failed in it. Manifest is 253, contract 70. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
The write-up claimed libsoup's digest authentication, evolution-data-server, gvfs and gnome-online-accounts all reach for GHmac. Three of those four are wrong, and the fourth is right for a different feature. Measured instead: All 163 runtime RPMs the factory currently publishes were unpacked and scanned for g_hmac_new, g_hmac_update and the g_compute_hmac_for_* family. Zero hits, with the scanner validated against Hummingbird's glib2, which does carry them. Of the desktop packages this branch adds but has not published, taken from Fedora's builds of the same sources, only libsoup3 references them -- evolution-data-server, gnome-online-accounts, glib-networking, gvfs and gnome-shell do not. And libsoup's single use is soup-auth-ntlm.c calling g_compute_hmac_for_data with G_CHECKSUM_MD5 for NTLMv2; digest authentication never touches GHmac. So the exposure is cockpit's authentication and NTLMv2, not the desktop stack. The reason to report it upstream is unchanged but restated honestly: the disablement is unconditional and silent, so g_hmac_new returns NULL on a machine that is not in FIPS mode and callers that do not check get a crash or a wrong answer rather than a policy error. Docs only -- docs/** is not in the workflow path filter, so this starts no build. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
The user's call, extending the previous one: not just anaconda-live out of the runtime contract, but Anaconda out of the manifest entirely. anaconda-webui was the only piece the factory still built -- the anaconda source package left this branch earlier, and anaconda-live is one of its subpackages, so with anaconda-webui gone nothing in the tree serves that path at all. anaconda-live stays in [unavailable] for the opposite reason it was put there. Before, it was excluded because the chain behind it could not be delivered; now it is excluded because nothing here answers for it, and without the exception the consumer transaction would ask for a package neither repository has. The comment says so rather than leaving the old reasoning in place. Also records the cockpit workaround that was investigated and not taken, since the research is worth more written down than repeated. Cockpit's only GHmac use is one function -- cockpit_auth_nonce in src/ws/cockpitauth.c, calling g_compute_hmac_for_data with G_CHECKSUM_SHA256 to mint CSRF and session tokens, a PRF rather than a protocol-mandated MAC. Cockpit already carries PKG_CHECK_MODULES(gnutls, [gnutls >= 3.6.0]), linked today only into src/tls, so a downstream patch could add gnutls to libcockpit_ws_a and swap that call for gnutls_hmac_fast(GNUTLS_MAC_SHA256, ...). No new dependency, same semantics, and the same move Fedora makes for GLib. Not taken because reviving cockpit revives the chain behind it, firefox included, for an installer Utah does not use. The docs entry for what the factory declines is rewritten to match: ffmpeg stays in scope on its own reasoning, and the Anaconda entry now covers the whole branch rather than cockpit alone. Manifest is 252, contract stays at 70. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #32, from reading the shard logs and the run history on this branch.
What the WebKitGTK logs actually show
The shards both succeeded (2h13m GTK 4, 2h52m GTK 3, against 5h01m for the old single job). But the compiler cache did nothing, and the reason is specific.
In-container statistics from job 100112414952:
Meanwhile ninja ran all 8955 targets through
/work/tools/sccache. Three separate faults:/work/tools/sccache.env. mozjs140's recipe does; webkitgtk's does not. Without it sccache has no backend configured and defaults to/root/.cache/sccache— inside the container, discarded when the container exits. Every object compiled, twice per run, thrown away.SCCACHE_IDLE_TIMEOUTdefaulted to 600s, so the server exited during the link and debuginfo phases. That is why the end-of-build statistics claim 7 requests for a build that made 8955.Changes
/workmount.sccache.envnow setsSCCACHE_DIR=/work/sccache,SCCACHE_CACHE_SIZE=6GandSCCACHE_IDLE_TIMEOUT=0. The lane restores that directory fromghcr.io/<owner>/utah-packages-ccache:<package>before the build and publishes it after."compiler_cache": truein the source manifest, currentlywebkitgtk,webkit2gtk4.1andmozjs140. Restoring and publishing gigabytes costs minutes; the 170 packages that build in under ten minutes would pay that for nothing.rebuild-lane.ymlis still hashed.The cache is published on failure too, and read on pull requests, for the same content-addressing reason: a partial compile leaves usable objects, and no build can look up an entry it would not itself have produced.
Expected effect
First run after this fills the cache. The run after that should cut both WebKitGTK shards well below two hours, and mozjs140's 20 minutes down to its link time.
Validation
tools/validate.py,tools/check_workflow_quoting.py, all three YAML files parse, and 20 unit tests pass including new coverage for the cache policy and for the spec sourcing the environment file. Not executed against a real runner — no rpm or podman here — so the PR run is the test.Scheduling note
Run 33608194980 is in flight on this branch. This PR is in a different concurrency group so it will not cancel it, and per the workflow's own comment concurrent waves cannot lose RPMs. It will queue jobs alongside that run; hold or close it if you would rather not share runners right now.
🤖 Generated with Claude Code
https://claude.ai/code/session_01KVSCoKWBWAuFFZyLwteEbB
Generated by Claude Code