Skip to content

Release chain silently skips publishing aiox-core: lockstep version sync is never persisted to main #827

Description

@Pedrovaleriolopez

Summary

Releasing v5.4.0 (PR #823) completed with every job green, but the legacy aiox-core npm package — the one users actually install (npm install -g aiox-core, npx aiox-core install) — was never republished. No red job, no error anywhere. Three distinct, compounding bugs in the same release chain were found and fixed live while investigating; documenting all of them here so they don't recur on the next release.

Bug 1 — lockstep version sync is never persisted to main

Symptom: npm view @aiox-squads/core version5.4.0 (correct, has the fix). npm view aiox-core version5.3.0 (stale, does not have the fix), with dependencies["@aiox-squads/core"] pinned exactly to 5.3.0.

Cause: .releaserc.json runs @semantic-release/execscripts/sync-version-lockstep.js in its prepareCmd, which updates .aiox-core/package.json and compat/aiox-core/package.json to the new version — but this only happens inside the Semantic Release job's own ephemeral checkout. There is no @semantic-release/git plugin, so the bump is never committed/pushed back to main. The downstream publish_legacy_aiox_core job runs in a separate workflow (npm-publish.yml, chained via gh workflow run dispatch) with its own fresh checkout of main, reads the still-5.3.0 compat/aiox-core/package.json, concludes "Version 5.3.0 already exists" → should_publish=false → silently skips.

Evidence:

  • Semantic Release run 31896753245 — log shows "The next release version is 5.4.0" and successful publish of @aiox-squads/core@5.4.0.
  • Chained npm-publish.yml run 31896804383publish_legacy_aiox_core job log: "Version 5.3.0 already exists for aiox-core" → skipped.
  • git show origin/main:compat/aiox-core/package.json (before fix) showed 5.3.0 despite the release having already shipped 5.4.0.

Fixed by: PR #825 — ran node scripts/sync-version-lockstep.js 5.4.0 (the pipeline's own tool, for byte-identical output) and committed .aiox-core/package.json + compat/aiox-core/package.json directly to main.

Bug 2 — the lockstep set is incomplete: root package.json isn't covered

Symptom: After fixing Bug 1 (PR #825), CI on main broke: tests/cli/validate-publish.test.js started failing. node bin/utils/validate-publish.js locally: "FAIL: version drift: .aiox-core/package.json version 5.4.0 does not match root package.json version 5.3.0. They must move in lockstep — root is SOT."

Cause: scripts/sync-version-lockstep.js treats root package.json as a read-only reference (it reads rootPkg.version to know the target, but never writes to root itself) — this is by design, documented in the script's own header comment. But root package.json is also never bumped anywhere else (same root cause as Bug 1: no @semantic-release/git plugin persists @semantic-release/npm's local version bump back to main). Before PR #825, root and .aiox-core/package.json were both stale at 5.3.0 — inconsistent with the tag, but internally consistent with each other, so the safety gate passed. PR #825 bumped only one side of that pairing, introducing a new drift that broke validate-aiox-core-namespace.js's root-as-SOT invariant. This was caught by the PR's own CI (Jest Tests, Validation Summary all red) but merged anyway with --admin before checking gh pr checks.

Fixed by: PR #826 — bumped root package.json to 5.4.0 (matching the already-published @aiox-squads/core@5.4.0 and the already-pushed v5.4.0 tag), synced package-lock.json's two version fields via npm install --package-lock-only, and regenerated .aiox-core/install-manifest.yaml. Verified locally (validate-publish.js → PASS, the specific Jest suite → 15/15) before pushing, and waited for full green CI before the --admin merge this time.

Bug 3 — publish_legacy_aiox_core's own if: never actually allows a scoped packages=aiox-core dispatch

Symptom: Re-dispatching npm-publish.yml -f packages=aiox-core (intending to publish only the legacy wrapper, without re-touching @aiox-squads/core) resulted in both publish and publish_legacy_aiox_core showing conclusion: skipped — nothing published.

Cause:

publish_legacy_aiox_core:
  needs: [test, build, publish]
  if: ${{ contains(...,'aiox-core,') && (needs.publish.result == 'success' || needs.publish.result == 'skipped') }}

The condition explicitly tries to allow the needs.publish.result == 'skipped' case (expected when only aiox-core — not core — is selected, since publish's own if: only runs on contains(packages, ',core,')). But GitHub Actions' default behavior overrides this: if a job in needs is skipped, the dependent job is also skipped by default, regardless of a custom if: expression, unless that expression is wrapped in always(). Since this if: doesn't call always(), publish_legacy_aiox_core can never run when publish was skipped — making a scoped packages=aiox-core-only dispatch a dead combination. It only worked (accidentally) in earlier attempts because packages=all was used, which makes publish run for real (not skip) and satisfies GitHub's default propagation rule as a side effect.

Workaround used today: dispatched with packages=all instead of packages=aiox-core. Safe because publish has its own internal idempotent "already published" check (same pattern seen for @aiox-squads/installer and aiox-core) — verified: it correctly no-op'd for @aiox-squads/core@5.4.0 (npm view @aiox-squads/core version unchanged at 5.4.0 after the run).

Suggested fix (not applied — flagging for review): add always() to the front of publish_legacy_aiox_core's if: (and audit the other conditional jobs in this workflow for the same pattern), so a scoped packages=aiox-core dispatch actually works as documented/intended.

Impact

Every future release using this chain repeats Bug 1/2 unless someone manually notices and intervenes (as happened here, live, mid-release). The scoped fix (Bug 3) means operators can't cleanly republish just the legacy wrapper without also re-touching the scoped package's publish job. In the concrete case that triggered all this: a student with a Pro-activation failure would have remained on a broken aiox-core@5.3.0 even after a "successful" v5.4.0 release, until this was manually caught and fixed today (PRs #825, #826, plus a manual packages=all re-dispatch of npm-publish.yml).

Secondary finding: Cross-Platform Tests (windows-latest) has been red on every push to main since at least 2026-07-10

Unrelated to the above, but noticed while confirming main's post-merge status: the push-triggered CI workflow's Cross-Platform Tests (windows-latest, Node 18/20/22/24) jobs have failed on every push-to-main run checked going back to commit 0ad5a9f3 (2026-07-10) — predates all of today's changes. This job doesn't run on pull_request events (shows skipping there), so it's invisible in normal PR review and has presumably been silently red for over a month.

Suggested remediation options (not prescribing — flagging for a maintainer decision)

  • Bug 1/2: either add a scoped @semantic-release/git plugin (committing just the version-bearing files back to main after a successful release), or have the release job directly trigger npm-publish.yml with an explicit version input instead of having downstream jobs re-derive it from a fresh (and potentially stale) checkout of main.
  • Bug 3: add always() to publish_legacy_aiox_core's if: condition (and audit sibling jobs for the same gap).
  • Process hygiene: should_publish=false / job-skipped states in this chain should never render as a fully green run without an explicit summary callout — worth a ::warning:: annotation at minimum.
  • Cross-Platform Tests: separately worth its own triage — it's been broken long enough that it's providing zero signal today.

Labels

Suggest: bug, infra, release, ci, high priority (this directly caused a real user-facing regression to go unpatched after a "successful" release).

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: cliCLI tools (bin/, packages/aios-pro-cli/)area: installerInstaller and setup (packages/installer/)ci/cdpriority: P1Critical — blocks users, immediate attentionstatus: needs-triageAwaiting initial triagetype: bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions