Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ __pycache__/
.venv/

# Build artifacts
dist/
dist/*
!dist/*.whl
htmlcov/
coverage.json
coverage.xml
Expand Down
3 changes: 3 additions & 0 deletions .github/actions/detect-changes/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,6 @@ runs:
- '.github/actions/**'
- '.mise.toml'
- '.mise/tasks/**'
- '.dockerignore'
- 'containers/Dockerfile.test_ci'
- 'tools/release_version.py'
18 changes: 12 additions & 6 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,21 +215,27 @@ Scans PRs for accidentally committed secrets. False positives can be added to `.

## Release Workflow (Production)

The production release workflow publishes to test PyPI and regular PyPI. It also creates release notes
The production release workflow verifies and publishes the wheel to Test PyPI
and PyPI, creates a GitHub release, and publishes versioned documentation for
final releases.

### How to Release

1. Push a tag to the repository (start with a release candidate like `v0.0.5rc0` for big changes)
2. Monitor the release pipeline to see it makes its way to Test PyPI/PyPI.
Follow the release preparation, candidate validation, promotion, and
post-release procedures in the [contributor guide](../../CONTRIBUTING.md#releasing).
The guide documents the read-only `release:prepare` helper and the supported
tag formats.

### Release Process

The workflow performs the following steps:

1. Build wheel - Builds the production wheel
2. Push to test PyPI
3. Publish to PyPI - Uploads to PyPI
4. Create GitHub release
2. Verify the wheel installs in a clean end-user container
3. Push to test PyPI
4. Publish to PyPI - Uploads to PyPI
5. Create GitHub release
6. Publish versioned documentation for final releases

## Reusable Workflows

Expand Down
43 changes: 38 additions & 5 deletions .github/workflows/ci-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,9 +54,9 @@ jobs:
src: ${{ steps.changes.outputs.src }}
tests: ${{ steps.changes.outputs.tests }}
pytest_ini: ${{ steps.changes.outputs.pytest_ini }}
pyproject: ${{ steps.changes.outputs.pyproject }}
uv_lock: ${{ steps.changes.outputs.uv_lock }}
deps: ${{ steps.changes.outputs.deps }}
docs_src: ${{ steps.changes.outputs.docs_src }}
ci: ${{ steps.changes.outputs.ci }}
any: ${{ steps.changes.outputs.any }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
Expand Down Expand Up @@ -119,6 +119,39 @@ jobs:
- name: Run ty type checks
run: mise run typecheck

wheel-install:
name: End-user Wheel Install
needs: changes
if: >-
${{
always() &&
(
github.event_name == 'workflow_dispatch' ||
needs.changes.outputs.src == 'true' ||
Comment thread
binaryaaron marked this conversation as resolved.
needs.changes.outputs.deps == 'true' ||
needs.changes.outputs.ci == 'true'
)
}}
runs-on: ubuntu-latest
steps:
- name: checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
fetch-depth: 0

- name: Setup Python environment
uses: ./.github/actions/setup-python-env
with:
checkout: "false"
bootstrap-tools: "true"

- name: Build wheel
run: mise run build-wheel

- name: Verify clean end-user wheel install
run: mise run release:verify-wheel

unit-test:
name: Unit Tests
needs: changes
Expand Down Expand Up @@ -191,8 +224,7 @@ jobs:
needs.changes.outputs.src == 'true' ||
needs.changes.outputs.tests == 'true' ||
needs.changes.outputs.pytest_ini == 'true' ||
needs.changes.outputs.uv_lock == 'true' ||
needs.changes.outputs.pyproject == 'true'
needs.changes.outputs.deps == 'true'
)
}}
runs-on: ubuntu-latest
Expand Down Expand Up @@ -223,14 +255,15 @@ jobs:
ci-status:
name: CI Status
if: always() && !cancelled()
needs: [changes, format, typecheck, unit-test, smoke-test]
needs: [changes, format, typecheck, wheel-install, unit-test, smoke-test]
runs-on: ubuntu-latest
steps:
- name: Check job results
run: |
echo "changes: ${{ needs.changes.result }}"
echo "format: ${{ needs.format.result }}"
echo "typecheck: ${{ needs.typecheck.result }}"
echo "wheel: ${{ needs.wheel-install.result }}"
echo "unit-test: ${{ needs.unit-test.result }}"
echo "smoke-test: ${{ needs.smoke-test.result }}"

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/container-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,8 @@ jobs:
type=raw,value=${{ matrix.variant }},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=raw,value=latest-${{ matrix.variant }},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=sha,prefix=sha-,suffix=-${{ matrix.variant }}
type=semver,pattern={{version}}-${{ matrix.variant }}
type=semver,pattern={{major}}.{{minor}}-${{ matrix.variant }}
type=pep440,pattern={{version}}-${{ matrix.variant }}
type=pep440,pattern={{major}}.{{minor}}-${{ matrix.variant }}
labels: |
org.opencontainers.image.version=${{ steps.package-version.outputs.version }}
com.nvidia.nemo.safe-synthesizer.extra=${{ matrix.extra }}
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,9 @@ jobs:
echo "is_prerelease=$IS_PRERELEASE" >> "$GITHUB_OUTPUT"
echo "Built: $WHEEL (version $VERSION)"

- name: Verify clean end-user wheel install
run: mise run release:verify-wheel

- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: nemo-safe-synthesizer-${{ steps.build.outputs.version }}
Expand Down
13 changes: 13 additions & 0 deletions .mise/tasks/release/prepare
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#MISE description="Prepare the next release tag from local Git tags without creating, deleting, or pushing tags."

set -euo pipefail
source "${MISE_CONFIG_ROOT}/.mise/tasks/_lib.sh"

tool_root="${MISE_CONFIG_ROOT:-$(git rev-parse --show-toplevel)}"
python_version="$(resolve_python_version)"

uv --no-config run --no-project --python "${python_version}" \
"${tool_root}/tools/release_version.py" "$@"
25 changes: 25 additions & 0 deletions .mise/tasks/release/verify-wheel
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#MISE description="Verify a built wheel with clean end-user CPU installation and CUDA dependency resolution."

set -euo pipefail
source "${MISE_CONFIG_ROOT}/.mise/tasks/_lib.sh"

tool_root="${MISE_CONFIG_ROOT:-$(git rev-parse --show-toplevel)}"
container_cmd="$(resolve_container_cmd)"

shopt -s nullglob
wheels=("${tool_root}"/dist/*.whl)
if (( ${#wheels[@]} != 1 )); then
echo "Error: expected exactly one wheel under ${tool_root}/dist, found ${#wheels[@]}" >&2
exit 1
fi

"${container_cmd}" build \
--platform "${CONTAINER_TEST_PLATFORM:-linux/amd64}" \
--progress=plain \
--file "${CONTAINER_TEST_FILE:-containers/Dockerfile.test_ci}" \
--target wheel-install \
--tag "${CONTAINER_WHEEL_INSTALL_IMAGE:-nss-wheel-install:latest}" \
"${tool_root}"
74 changes: 63 additions & 11 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -685,14 +685,14 @@ Before contributing, run `mise run format` and `mise run check`. See `AGENTS.md`

Pushing a `v*` tag starts two workflows. [`release.yml`](.github/workflows/release.yml)
publishes the wheel to Test PyPI and PyPI, creates a GitHub release, and
publishes versioned documentation for stable releases.
publishes versioned documentation for final releases, including post-releases.
[`container-build.yml`](.github/workflows/container-build.yml) publishes the
CUDA image to GitHub Container Registry (GHCR).

Release versions follow [PEP 440](https://peps.python.org/pep-0440/) with major,
minor, and patch release numbers. This project uses stable releases and release
candidates only; prerelease versions append `rcN` without a dash. The GitHub tag
always starts with a `v` prefix.
minor, and patch release numbers. This project uses stable releases, release
candidates, and post-releases. Prerelease versions append `rcN` without a dash;
post-releases append `.postN`. The GitHub tag always starts with a `v` prefix.

Examples:

Expand All @@ -702,22 +702,47 @@ Examples:
| `v2.1.3` | `2.1.3` | ✅ |
| `v0.0.5rc0` | `0.0.5rc0` | ✅ |
| `v0.1.2rc5` | `0.1.2rc5` | ✅ |
| `v0.1.6.post1` | `0.1.6.post1` | ✅ |
| `1.0.0` | | ❌ No `v` prefix |
| `release-1.0` | | ❌ Wrong format |
| `v0.0.7-rc4` | | ❌ Dash before rc suffix |
| `v0.1.6-post1` | | ❌ Dash before post-release suffix |
| `v0.1.3a1` | | ❌ Alpha prereleases are not used; use rcN only |

### Release Preparation Helper

Run `mise run release:prepare -- [OPTIONS]` to compute a release tag before
creating it. The helper reads local Git tags and resolves the requested target
commit, but it does not create, delete, or push tags.

- `--bump major`, `--bump minor`, and `--bump patch` propose the corresponding
next version's initial `rc0` tag. The default is `patch`.
- `--bump post` proposes the next `.postN` tag for the latest stable version.
- `--ref REF` selects the commit to tag and defaults to `HEAD`.
- `--json` emits the computed release plan as machine-readable JSON.

Fetch the tags you intend the helper to consider before running it. It rejects
malformed release tags, post-release tags without their stable base tag, and an
initial `rc0` proposal when an RC already exists for that version.

### Release Checklist

#### Before Publishing

- Fetch `origin/main` and tags, choose the exact release commit, and confirm its
normal CI and manually dispatched GPU Tests run passed.
- Choose unused candidate and stable tags, then record the candidate's exact
`origin/main` SHA.
- Choose the unused tag or tags required for the release type, then record the
exact `origin/main` SHA.
- Decide whether GHCR visibility or a separate nSpect or Pulse scan blocks the
release. Those scans are not part of the GitHub release workflows.

After fetching tags, preview the next release's initial `rc0` version and
resolve its target commit without creating or pushing a tag:

```bash
mise run release:prepare -- --ref origin/main
```

#### Tag Trigger

Create the candidate tag at the recorded SHA and push it. This automatically
Expand Down Expand Up @@ -778,21 +803,48 @@ git tag "${STABLE_TAG}" "${RELEASE_SHA}"
git push origin "refs/tags/${STABLE_TAG}"
```

#### After Publishing Stable
#### Post-release

Use a post-release for a packaging or release correction that does not warrant
a new regular patch version. A post-release is final, so it does not use the
release-candidate promotion sequence. Preview the next post-release tag and
resolve its target commit without creating or pushing a tag:

```bash
mise run release:prepare -- --bump post --ref origin/main
```

After reviewing the output, create the proposed tag at the resolved commit and
push it. For example:

```bash
POST_TAG=v0.1.6.post1
RELEASE_SHA="$(git rev-parse 'origin/main^{commit}')"
git tag "${POST_TAG}" "${RELEASE_SHA}"
git push origin "refs/tags/${POST_TAG}"
```

The container workflow publishes `X.Y.Z.postN-cu129`, the immutable
`sha-<short-sha>-cu129` tag, and the mutable `cu129` and `latest-cu129` aliases.
PEP 440 post-releases do not move the shortened `X.Y-cu129` tag. Validate the
post-release wheel and immutable container tag before announcing the release.

#### After Publishing a Final Release

After publishing:

- Verify both tag-triggered workflows passed at the tested SHA.
- Confirm Test PyPI and production PyPI contain the stable version.
- Confirm Test PyPI and production PyPI contain the published version.
- Confirm the GitHub release is not marked as a prerelease.
- Confirm versioned documentation is available at
`https://nvidia-nemo.github.io/Safe-Synthesizer/<version>/`.
- Confirm GHCR exposes the stable `X.Y.Z-cu129` and `X.Y-cu129` tags with the
intended visibility.
- Confirm GHCR exposes the expected tags with the intended visibility. Regular
stable releases publish `X.Y.Z-cu129` and `X.Y-cu129`; post-releases publish
`X.Y.Z.postN-cu129` without moving `X.Y-cu129`.
- Coordinate a NeMo Platform package or container pin, documentation update,
and downstream release when Platform should consume the new version. This is
not currently automated by the Safe Synthesizer release workflow.
- Announce the release only after artifacts and stable documentation pass
- Announce the release only after artifacts and versioned documentation pass
verification.

## NMP Integration
Expand Down
35 changes: 35 additions & 0 deletions containers/Dockerfile.test_ci
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,41 @@ RUN MISE_GPG_KEY=24853EC9F655CE80B48E6C3A8B81C9D17413A06D \
MISE_YES=1 mise trust && \
PYTHON_VERSION="${PYTHON_VERSION}" MISE_YES=1 mise run setup

FROM setup AS wheel-install

ENV UV_LINK_MODE=copy

COPY dist/*.whl /tmp/dist/

RUN --mount=type=cache,target=/root/.cache/uv \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish Dockerfiles in github had syntax highlighting for these

my nit would be to put this section into a script since it's a bit rough to read here

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 25671818. The CI-facing orchestration now lives in mise run release:verify-wheel; the actual install and smoke checks remain in the wheel-install stage so the Docker build itself verifies the clean end-user environment.

wheel_count="$(find /tmp/dist -maxdepth 1 -type f -name '*.whl' | wc -l)" && \
test "${wheel_count}" -eq 1 && \
wheel="$(find /tmp/dist -maxdepth 1 -type f -name '*.whl')" && \
uv --no-config venv --clear --python /usr/local/bin/python /opt/wheel-cpu && \
uv --no-config pip install \
--python /opt/wheel-cpu/bin/python \
--no-sources \
--default-index https://pypi.org/simple \
--index https://download.pytorch.org/whl/cpu \
--index-strategy unsafe-best-match \
"nemo-safe-synthesizer[cpu,engine] @ file://${wheel}" && \
uv --no-config pip check --python /opt/wheel-cpu/bin/python && \
/opt/wheel-cpu/bin/python -c \
"from importlib.metadata import version; from nemo_safe_synthesizer.package_info import __version__; assert __version__ == version('nemo-safe-synthesizer')" && \
/opt/wheel-cpu/bin/safe-synthesizer --help >/dev/null && \
uv --no-config venv --clear --python /usr/local/bin/python /opt/wheel-cu129 && \
uv --no-config pip install \
--python /opt/wheel-cu129/bin/python \
--no-sources \
--dry-run \
--default-index https://pypi.org/simple \
--index https://flashinfer.ai/whl/cu129 \
--index https://download.pytorch.org/whl/cu129 \
--index https://wheels.vllm.ai/ee0da84ab9e04ac7610e28580af62c365e898389/cu129 \
--index-strategy unsafe-best-match \
"nemo-safe-synthesizer[cu129,engine] @ file://${wheel}" && \
rm -rf /opt/wheel-cpu /opt/wheel-cu129

FROM setup AS install-deps
# Install Python dependencies (cached until pyproject.toml, uv.lock, or src/ or tests/ change)
COPY pyproject.toml .
Expand Down
13 changes: 10 additions & 3 deletions containers/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,16 +123,22 @@ before enabling a platform in CI.
## CPU Test Image

`Dockerfile.test_ci` provides a CPU-only image for running unit tests locally
or in CI without a GPU. It uses a two-stage build: `setup` (system packages +
mise-managed tools) and `install-deps` (Python environment via
`mise run bootstrap-nss cpu`).
or in CI without a GPU. Its `setup` stage installs system packages and
mise-managed tools, while `install-deps` creates the Python environment with
`mise run bootstrap-nss cpu`. The separate `wheel-install` stage installs the
built wheel without project sources, runs CPU package and CLI checks, and
resolves the CUDA dependency set.

```bash
# Run CI unit tests in a container
mise run test:ci-container

# Verify mise-managed tools install correctly (fast -- setup stage only)
mise run test:tool-install

# Verify the built wheel in a clean container stage
mise run build-wheel
mise run release:verify-wheel
```

### CPU Test Mise Tasks
Expand All @@ -143,3 +149,4 @@ mise run test:tool-install
| `container:build:test-setup` | Build only the setup stage (tools, no Python deps) |
| `test:ci-container` | Build and run CI unit tests |
| `test:tool-install` | Verify mise-managed tools install correctly (setup stage only) |
| `release:verify-wheel` | Install and verify the built wheel in the clean wheel stage |
Loading
Loading