Repository navigation
ci: stop the publish pipeline breaking at release time - #1924
Conversation
v1.12.0 reached PyPI as nothing at all. The Windows wheel failed on a runner image that rolled under us -- MSVC 14.51, Visual Studio 18 -- with no commit from this repo, and publish needs the whole matrix, so five working wheels and a working sdist were withheld too. v1.11.0 had built the same tree two weeks earlier. Patching the symptom leaves the next roll to do it again, so this addresses the four things that made a runner change into a failed release. 1. Pinned runner images across every publish workflow. ubuntu-latest, windows-latest and friends move when GitHub moves them; a toolchain upgrade should arrive as a reviewed PR that can fail in CI, not as a surprise during a release. ubuntu-latest is 24.04 today, so that half is a no-op now and a guard later. Windows pins to windows-2022, which is the image that worked. 2. sdk/python/Cargo.lock is committed. That directory is its own workspace, excluded from the root one, so the root lockfile never constrained it and an ignored lock meant the native build re-resolved every run: cc moved 1.6.0 -> 1.9.0 between the two releases with nothing in git to show it. Wheels now build --locked, so a wheel is reproducible from its tag. 3. publish no longer requires an all-green matrix. One platform failing used to withhold every artifact, which is a self-inflicted outage: five wheels help five platforms and the sdist is the floor for the sixth. skip-existing already makes a backfill run safe, so a partial upload is a release that can be completed rather than one that has to be redone. A new matrix-complete job keeps the run red, and the publish summary names every platform a user will not find a wheel for. 4. The matrix runs weekly, publishing nothing. It previously ran only at release, on a tag, after the tag existed -- so every break was found at the one moment when the only options are a hotfix or a retag. A Tuesday build turns that into ordinary CI noise. Verified: dispatched against the v1.12.0 tree, wheel (windows-x64) succeeded where the release run failed (run 39257982228, before 1-4; the sccache change is PR #1923).
#1923 landed the per-platform sccache field this branch was written against. Both sides edited the same maturin step: the resolution keeps #1923's sccache: ${{ matrix.sccache }} and this branch's --locked, and notes that the runner is now pinned to windows-2022, so sccache would probably work again and stays off regardless -- the pin will have to move eventually.
rustls-webpki 0.103.15 -> 0.103.16 (RUSTSEC-2026-0382) webpki-root-certs 1.0.9 -> 1.0.10 (RUSTSEC-2026-0383) Both are CVE-2026-111943: certificate verification may accept distrusted intermediates, and the bundled root certificates allow trust in distrusted emSign intermediates. The advisories' patched ranges are >= 0.103.16 and >= 1.0.10, which is what cargo resolves to within semver -- no manifest change is needed. These were published after this repo last ran CI on main (2026-10-19), and cargo-deny fetches the advisory database on every run, so the whole PR lane started failing supply-chain with nothing in the tree having changed. This is the ONE commit that unblocks it; #1924 was red for this and not for its own contents. cargo update -p for the two crates specifically rather than a bare cargo update: 5 insertions, 5 deletions, nothing else moved. (cherry picked from commit c704cbc)
rustls-webpki 0.103.15 -> 0.103.16 (RUSTSEC-2026-0382) webpki-root-certs 1.0.9 -> 1.0.10 (RUSTSEC-2026-0383) Both are CVE-2026-111943: certificate verification may accept distrusted intermediates, and the bundled root certificates allow trust in distrusted emSign intermediates. The advisories' patched ranges are >= 0.103.16 and >= 1.0.10, which is what cargo resolves to within semver -- no manifest change is needed. These were published after this repo last ran CI on main (2026-10-19), and cargo-deny fetches the advisory database on every run, so the whole PR lane started failing supply-chain with nothing in the tree having changed. This is the ONE commit that unblocks it; #1924 was red for this and not for its own contents. cargo update -p for the two crates specifically rather than a bare cargo update: 5 insertions, 5 deletions, nothing else moved. (cherry picked from commit c704cbc)
Kalyanpulikanti-01
left a comment
There was a problem hiding this comment.
Local PR agent — AI review of exact head 7642d94.
Verdict: changes_requested.
Reviewed all six changed files. Two workflow regressions affect release backfills and non-engine releases.
-
P2 .github/workflows/publish-pypi.yml:115 — Unconditionally adding
--lockedbreaks the existinginputs.refrecovery path for tags created before this PR: checkout retrieves that tag, which lacks the newly committed Python Cargo.lock, so every wheel fails because Cargo cannot create a lockfile under--locked. The successful sdist can then be published alone. Preserve recovery for older tags through an explicit legacy lockfile strategy, while enforcing the committed lockfile for new releases. -
P2 .github/workflows/publish-pypi.yml:298 —
!cancelled()also runs this job for non-v data releases, even though wheels, sdist, and publish are deliberately skipped. The check seessdist=skippedand fails, turning every such release into a failed Python publishing run. Apply the same engine-release event guard used by the build jobs, while retaining execution after actual build failures.
The queue is held here for a maintainer; no automatic code edits or merge.
This is an AI review comment, not a human approval or a CI result.
Kalyanpulikanti-01
left a comment
There was a problem hiding this comment.
Reviewed all six changed files and checked the standalone Python Cargo manifest and lockfile against dependency resolution and native build results. Legacy backfills preserve the original manifest only when the source predates the lockfile policy; current sources remain locked. Publication-condition and completion-shell checks confirm that partial wheel failure allows available artifacts to publish while keeping the run red, and non-engine releases skip the completion gate. Merge must wait for CI and both complete five-platform build-and-test matrices.
v1.12.0 reached PyPI as nothing at all. The Windows wheel failed on a runner image that rolled under us — MSVC 14.51 / Visual Studio 18 — with no commit from this repo, and
publishneeds the whole matrix, so five working wheels and a working sdist were withheld with it. v1.11.0 had built the same tree two weeks earlier.#1923 is the unblock. This is the part that stops the next roll doing it again, because patching each break as it happens is not a fix — it just moves the next outage to the next release. Four causes:
1. Pinned runner images, everywhere a publish workflow runs
ubuntu-latest/windows-latestmove when GitHub moves them. A toolchain upgrade should arrive as a reviewed PR that can fail in CI, not mid-release. Windows pins towindows-2022, the image that worked.ubuntu-latestis 24.04 today, so that half is a no-op now and a guard later.2.
sdk/python/Cargo.lockis committed, and wheels build--lockedThat directory is its own workspace,
excluded from the root one, so the root lockfile never constrained it — and the lock was gitignored, so the native build re-resolved on every run.ccwent 1.6.0 to 1.9.0 between the two releases with nothing in git to show it. A published wheel should be reproducible from the tag that built it. (cargo metadata --lockedverified against the manifest.)3.
publishno longer requires an all-green matrixThis is the defect that turned one platform's problem into a total outage. Five wheels help five platforms, and the sdist is the floor for the sixth; publishing nothing helps nobody.
skip-existing: truewas already set, so a backfill run tops up the missing files without touching what landed — a partial upload is a release that can be completed rather than one that must be redone.Two things keep a partial release from looking clean:
matrix-completejob fails after the upload, so the run is red and attached to the release that is actually incomplete;This is a policy change and the one judgement call here — if you would rather a release be all-or-nothing, say so and I will drop item 3; the other three stand on their own.
4. The matrix runs weekly, publishing nothing
It previously ran only at release, on a tag, after the tag existed — so every break was discovered at the one moment when the options are a hotfix or a retag.
schedule: 17 5 * * 2builds all six platforms and uploads nothing (publishis already gated onreleaseor an explicittarget), turning this class of failure into ordinary CI noise on a Tuesday.Not covered
The weekly build deliberately does not include the Docker image: that build takes 3h18m (measured on this release), and a weekly run of it is a real cost to weigh separately. Its runners are pinned here, so it gets item 1 but not item 4. Say if you want it scheduled monthly.
crates.io's
samyama/samyama-sdkgap is unrelated and unaffected — Trusted Publishing cannot create a new crate, which needs a one-time manual publish, not a workflow change.