Skip to content

ci: stop the publish pipeline breaking at release time - #1924

Merged
Kalyanpulikanti-01 merged 5 commits into
mainfrom
ci/harden-publish-pipeline
Oct 22, 2026
Merged

Kalyanpulikanti-01 merged 5 commits into
mainfrom
ci/harden-publish-pipeline

Conversation

@Kalyanpulikanti-01

Copy link
Copy Markdown
Contributor

v1.12.0 reached PyPI as nothing at all. The Windows wheel failed on a runner image that rolled under us — MSVC 14.51 / Visual Studio 18 — with no commit from this repo, and publish needs the whole matrix, so five working wheels and a working sdist were withheld with it. v1.11.0 had built the same tree two weeks earlier.

#1923 is the unblock. This is the part that stops the next roll doing it again, because patching each break as it happens is not a fix — it just moves the next outage to the next release. Four causes:

1. Pinned runner images, everywhere a publish workflow runs

ubuntu-latest / windows-latest move when GitHub moves them. A toolchain upgrade should arrive as a reviewed PR that can fail in CI, not mid-release. Windows pins to windows-2022, the image that worked. ubuntu-latest is 24.04 today, so that half is a no-op now and a guard later.

2. sdk/python/Cargo.lock is committed, and wheels build --locked

That directory is its own workspace, excluded from the root one, so the root lockfile never constrained it — and the lock was gitignored, so the native build re-resolved on every run. cc went 1.6.0 to 1.9.0 between the two releases with nothing in git to show it. A published wheel should be reproducible from the tag that built it. (cargo metadata --locked verified against the manifest.)

3. publish no longer requires an all-green matrix

This is the defect that turned one platform's problem into a total outage. Five wheels help five platforms, and the sdist is the floor for the sixth; publishing nothing helps nobody. skip-existing: true was already set, so a backfill run tops up the missing files without touching what landed — a partial upload is a release that can be completed rather than one that must be redone.

Two things keep a partial release from looking clean:

  • a new matrix-complete job fails after the upload, so the run is red and attached to the release that is actually incomplete;
  • the publish summary names every platform a user will not find a wheel for, with the command to top it up.

This is a policy change and the one judgement call here — if you would rather a release be all-or-nothing, say so and I will drop item 3; the other three stand on their own.

4. The matrix runs weekly, publishing nothing

It previously ran only at release, on a tag, after the tag existed — so every break was discovered at the one moment when the options are a hotfix or a retag. schedule: 17 5 * * 2 builds all six platforms and uploads nothing (publish is already gated on release or an explicit target), turning this class of failure into ordinary CI noise on a Tuesday.

Not covered

The weekly build deliberately does not include the Docker image: that build takes 3h18m (measured on this release), and a weekly run of it is a real cost to weigh separately. Its runners are pinned here, so it gets item 1 but not item 4. Say if you want it scheduled monthly.

crates.io's samyama / samyama-sdk gap is unrelated and unaffected — Trusted Publishing cannot create a new crate, which needs a one-time manual publish, not a workflow change.

v1.12.0 reached PyPI as nothing at all. The Windows wheel failed on a runner
image that rolled under us -- MSVC 14.51, Visual Studio 18 -- with no commit
from this repo, and publish needs the whole matrix, so five working wheels and
a working sdist were withheld too. v1.11.0 had built the same tree two weeks
earlier. Patching the symptom leaves the next roll to do it again, so this
addresses the four things that made a runner change into a failed release.

1. Pinned runner images across every publish workflow. ubuntu-latest,
   windows-latest and friends move when GitHub moves them; a toolchain
   upgrade should arrive as a reviewed PR that can fail in CI, not as a
   surprise during a release. ubuntu-latest is 24.04 today, so that half is a
   no-op now and a guard later. Windows pins to windows-2022, which is the
   image that worked.

2. sdk/python/Cargo.lock is committed. That directory is its own workspace,
   excluded from the root one, so the root lockfile never constrained it and
   an ignored lock meant the native build re-resolved every run: cc moved
   1.6.0 -> 1.9.0 between the two releases with nothing in git to show it.
   Wheels now build --locked, so a wheel is reproducible from its tag.

3. publish no longer requires an all-green matrix. One platform failing used
   to withhold every artifact, which is a self-inflicted outage: five wheels
   help five platforms and the sdist is the floor for the sixth. skip-existing
   already makes a backfill run safe, so a partial upload is a release that
   can be completed rather than one that has to be redone. A new
   matrix-complete job keeps the run red, and the publish summary names every
   platform a user will not find a wheel for.

4. The matrix runs weekly, publishing nothing. It previously ran only at
   release, on a tag, after the tag existed -- so every break was found at the
   one moment when the only options are a hotfix or a retag. A Tuesday build
   turns that into ordinary CI noise.

Verified: dispatched against the v1.12.0 tree, wheel (windows-x64) succeeded
where the release run failed (run 39257982228, before 1-4; the sccache change
is PR #1923).
#1923 landed the per-platform sccache field this branch was written against.
Both sides edited the same maturin step: the resolution keeps #1923's
sccache: ${{ matrix.sccache }} and this branch's --locked, and notes that the
runner is now pinned to windows-2022, so sccache would probably work again and
stays off regardless -- the pin will have to move eventually.
nareshkolimi pushed a commit that referenced this pull request Oct 21, 2026
rustls-webpki 0.103.15 -> 0.103.16  (RUSTSEC-2026-0382)
webpki-root-certs 1.0.9 -> 1.0.10   (RUSTSEC-2026-0383)

Both are CVE-2026-111943: certificate verification may accept distrusted
intermediates, and the bundled root certificates allow trust in distrusted
emSign intermediates. The advisories' patched ranges are >= 0.103.16 and
>= 1.0.10, which is what cargo resolves to within semver -- no manifest change
is needed.

These were published after this repo last ran CI on main (2026-10-19), and
cargo-deny fetches the advisory database on every run, so the whole PR lane
started failing supply-chain with nothing in the tree having changed. This is
the ONE commit that unblocks it; #1924 was red for this and not for its own
contents.

cargo update -p for the two crates specifically rather than a bare
cargo update: 5 insertions, 5 deletions, nothing else moved.

(cherry picked from commit c704cbc)
nareshkolimi pushed a commit that referenced this pull request Oct 21, 2026
rustls-webpki 0.103.15 -> 0.103.16  (RUSTSEC-2026-0382)
webpki-root-certs 1.0.9 -> 1.0.10   (RUSTSEC-2026-0383)

Both are CVE-2026-111943: certificate verification may accept distrusted
intermediates, and the bundled root certificates allow trust in distrusted
emSign intermediates. The advisories' patched ranges are >= 0.103.16 and
>= 1.0.10, which is what cargo resolves to within semver -- no manifest change
is needed.

These were published after this repo last ran CI on main (2026-10-19), and
cargo-deny fetches the advisory database on every run, so the whole PR lane
started failing supply-chain with nothing in the tree having changed. This is
the ONE commit that unblocks it; #1924 was red for this and not for its own
contents.

cargo update -p for the two crates specifically rather than a bare
cargo update: 5 insertions, 5 deletions, nothing else moved.

(cherry picked from commit c704cbc)

@Kalyanpulikanti-01 Kalyanpulikanti-01 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Local PR agent — AI review of exact head 7642d94.

Verdict: changes_requested.

Reviewed all six changed files. Two workflow regressions affect release backfills and non-engine releases.

  • P2 .github/workflows/publish-pypi.yml:115 — Unconditionally adding --locked breaks the existing inputs.ref recovery path for tags created before this PR: checkout retrieves that tag, which lacks the newly committed Python Cargo.lock, so every wheel fails because Cargo cannot create a lockfile under --locked. The successful sdist can then be published alone. Preserve recovery for older tags through an explicit legacy lockfile strategy, while enforcing the committed lockfile for new releases.

  • P2 .github/workflows/publish-pypi.yml:298 — !cancelled() also runs this job for non-v data releases, even though wheels, sdist, and publish are deliberately skipped. The check sees sdist=skipped and fails, turning every such release into a failed Python publishing run. Apply the same engine-release event guard used by the build jobs, while retaining execution after actual build failures.

The queue is held here for a maintainer; no automatic code edits or merge.

This is an AI review comment, not a human approval or a CI result.

@Kalyanpulikanti-01 Kalyanpulikanti-01 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all six changed files and checked the standalone Python Cargo manifest and lockfile against dependency resolution and native build results. Legacy backfills preserve the original manifest only when the source predates the lockfile policy; current sources remain locked. Publication-condition and completion-shell checks confirm that partial wheel failure allows available artifacts to publish while keeping the run red, and non-engine releases skip the completion gate. Merge must wait for CI and both complete five-platform build-and-test matrices.

@Kalyanpulikanti-01
Kalyanpulikanti-01 merged commit 7b3b833 into main Oct 22, 2026
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant