Skip to content

fix(release): make the release checklist able to fail #840

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andyne13 wants to merge 1 commit into
base: develop
Choose a base branch
from
Open

fix(release): make the release checklist able to fail #840

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 92 additions & 11 deletions .github/RELEASING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,25 @@ Set the version once, and define the digest helper every step below uses:

```bash
export VER=v2.1.0 # the tag you just pushed
dg() { docker buildx imagetools inspect --raw "$1" 2>/dev/null | sha256sum | awk '{print "sha256:"$1}'; }

# Resolve a tag's manifest digest. Returns non-zero and prints nothing when the
# tag does not exist — do NOT pipe inspect straight into sha256sum: on a failed
# lookup it hashes empty input and returns
# sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855,
# a real-looking digest. Steps 2-4 would then report a missing image as present,
# and two missing tags would compare equal and pass.
dg() {
local raw
raw=$(docker buildx imagetools inspect --raw "$1" 2>/dev/null) || return 1
[ -n "$raw" ] || return 1
printf '%s' "$raw" | sha256sum | awk '{print "sha256:"$1}'
}
```

Check the helper itself before trusting it — this must print `MISSING`:

```bash
dg ghcr.io/linagora/openrag:v0.0.0-does-not-exist || echo MISSING
```

> **Why `--raw | sha256sum` and not `--format '{{.Manifest.Digest}}'`:** buildx
Expand Down Expand Up @@ -61,10 +79,19 @@ gh run view "$RUN_ID" --json jobs \
**FAIL** on any `skipped` — that is the v2.0.1 bug recurring. A hard gate:

```bash
gh run view "$RUN_ID" --json jobs --jq '[.jobs[] | select(.conclusion != "success")] | length'
# must print 0
bad=$(gh run view "$RUN_ID" --json jobs \
--jq '[.jobs[] | select(.conclusion != "success")] | length') || exit 1
if [ "$bad" -ne 0 ]; then
echo "FAIL: $bad job(s) did not conclude success — do not continue" >&2
exit 1
fi
echo "OK: every job concluded success"
Comment on lines +82 to +88

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Require all four release jobs before continuing.

This gate checks only the conclusions of returned jobs. If a required job is absent, bad is still 0 when the remaining jobs succeed. Assert that verify-tag, build-and-push-image, build-and-push-image-ray, and build-and-push-image-admin-ui are present before checking their conclusions.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/RELEASING.md around lines 82 - 88, Update the release-job validation
around the bad count to first verify that verify-tag, build-and-push-image,
build-and-push-image-ray, and build-and-push-image-admin-ui are all present in
the gh run view result. Fail the gate when any required job is missing, then
retain the existing non-success conclusion check before continuing.

```

Written as a gate, not a print: a command that only reports the count still
exits 0 when the count is non-zero, so a release could continue straight past a
skipped build job — the very thing this step exists to stop.

If `verify-tag` failed loudly, the tag is not an ancestor of `origin/main` —
fix the tag placement, do not rerun.

Expand Down Expand Up @@ -110,7 +137,13 @@ back-filled from a different build.
for pair in "ghcr.io/linagora/openrag linagoraai/openrag" \
"ghcr.io/linagora/openrag-admin-ui linagoraai/openrag-admin-ui"; do
set -- $pair; a=$(dg "$1:$VER"); b=$(dg "$2:$VER")
[ "$a" = "$b" ] && echo "OK $1 == $2" || echo "MISMATCH $1=$a $2=$b"
# The -n guards matter: without them two MISSING tags are both empty, compare
# equal, and print OK.
if [ -n "$a" ] && [ -n "$b" ] && [ "$a" = "$b" ]; then
echo "OK $1 == $2"
else
echo "MISMATCH $1=${a:-MISSING} $2=${b:-MISSING}"
fi
done
```

Expand All @@ -120,12 +153,21 @@ done

Steps 2–4 read metadata. This proves the bytes are actually fetchable.

`RepoDigests` entries are `repo@sha256:…`, while `dg` returns a bare
`sha256:…` — strip the repository prefix before comparing, or the two can never
match literally.

```bash
docker pull "linagoraai/openrag:$VER"
docker image inspect "linagoraai/openrag:$VER" --format '{{index .RepoDigests 0}}'
pulled=$(docker image inspect "linagoraai/openrag:$VER" \
--format '{{index .RepoDigests 0}}' | cut -d@ -f2)
registry=$(dg "linagoraai/openrag:$VER") || { echo "FAIL: tag not in registry" >&2; exit 1; }
[ "$pulled" = "$registry" ] \
Comment on lines 161 to +165

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== file excerpt =="
sed -n '140,175p' .github/RELEASING.md 2>/dev/null || true

echo "== nearby command context =="
git show HEAD:.github/RELEASING.md 2>/dev/null | sed -n '140,175p' || true

echo "== repo docker/digest refs =="
rg -n "docker (pull|image inspect)|RepoDigests|linagoraai/openrag|registry|verify" .github/RELEASING.md README.md 2>/dev/null || true

Repository: linagora/openrag

Length of output: 5259

🌐 Web query:

Docker CLI image inspect RepoDigests multiple entries index first entry repository may not match pulled image documentation

💡 Result:

The RepoDigests field in Docker image metadata is a list of content-addressable digests of image manifests from which a local image was referenced [1]. Because this field is a collection of all registry/repository locations from which an image has been pulled or pushed, it can contain multiple entries if the same image ID exists under different repository names or registry paths [2][3]. It is important to understand the following regarding RepoDigests: 1. Order and Indexing: The order of entries in the RepoDigests list is not guaranteed to align with the specific image tag or repository used during a recent pull [2]. Attempting to retrieve a specific entry using index 0 (e.g., {{index.RepoDigests 0}}) is unreliable if multiple digests are present, as the first entry may not correspond to the image name you used to pull the image [2]. 2. Nature of the Field: RepoDigests are attributes of where an image is stored rather than immutable properties of the image itself [3]. They record the source manifest digest at the time of the pull operation [3]. Consequently, the repository portion of the digest string is derived from the registry/repository path used during that pull and may not match the repository name you are currently working with if the image was pulled from or pushed to multiple locations [2][3]. 3. deduplication: Docker has implemented fixes in recent versions to prevent duplicate entries from appearing in the RepoDigests list, ensuring that each unique reference is stored only once [4][5][6]. To reliably work with these values, avoid relying on positional indexing. If you need a specific digest for a particular repository, you should filter or iterate through the list using external tools (such as grep or platform-specific template logic) to match the desired registry path string, rather than assuming the first element is the correct one [2].

Citations:

🌐 Web query:

Docker CLI image pull returns non-zero when pull fails separate from image inspect caching documented

💡 Result:

The Docker CLI docker pull command is designed to return a non-zero exit code when a pull operation fails [1]. This behavior is generally consistent with command-line standards where failures are signaled through exit status [2][3]. While some legacy reports or specific environmental configurations (such as certain uses of docker-compose) have occasionally raised questions about suppression of error codes [1], the docker pull command itself executes via the Docker daemon, which propagates errors returned during the pull process [4]. If the daemon encounters an error—such as an image not found, an authentication failure, or a network issue—the CLI client receives and reports this error, typically resulting in a non-zero exit status [5][4][1]. It is important to note that the Docker CLI's error handling and exit codes are not always granularly documented for every failure scenario [6]. In practice: - A failed docker pull due to registry unavailability, missing images, or access issues will generally result in a non-zero exit code [5][1]. - Some users have historically requested specific or distinct exit codes for different types of failures (e.g., distinguishing between authentication issues and network issues) [5][2], but these are not currently implemented as a standardized, stable API, and users should rely on the standard non-zero exit convention to detect failure [2]. - Behaviors may occasionally appear inconsistent across different versions or when used in combination with tools like docker-compose, which may have its own logic for handling or suppressing exit statuses [1]. There is no documented "caching" mechanism that would cause a failed pull to return a zero exit code under normal circumstances; if an image is already cached locally (pulled successfully previously), docker pull will verify its existence and return zero, which is the expected "success" behavior. If you are experiencing unexpected zero exit codes during a failed pull, ensure you are not masking the error in a shell script (e.g., using || true or checking the exit code of the wrong process) [2].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
text = Path(".github/RELEASING.md").read_text()
lines = text.splitlines()
print("contains ignored pull:", "docker pull" in text and "||" in "\n".join(lines[158:164]))
print("line 161-165:")
for i in range(157, min(166, len(lines))):
    print(f"{i+1}: {lines[i]}")
PY

if command -v docker >/dev/null 2>&1; then
  echo "== local Docker availability =="
  docker version 2>&1 | sed -n '1,20p'
else
  echo "== local Docker availability: not installed =="
fi

Repository: linagora/openrag

Length of output: 661

Fail on Docker pull errors and compare only the matching repository digest.

docker pull status is ignored, so the script can inspect an older locally cached image after a failed pull and may still pass if that stale digest equals the registry value. Also avoid {{index .RepoDigests 0}} because RepoDigests is an unordered collection. Use the pull exit status, select only the linagoraai/openrag entry, reject an empty digest, then compare with registry.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/RELEASING.md around lines 161 - 165, Update the Docker verification
flow in .github/RELEASING.md to fail immediately when docker pull fails,
preventing inspection of a stale cached image. When extracting pulled, select
the RepoDigests entry for linagoraai/openrag rather than indexing the unordered
collection, reject an empty digest, then compare that digest with registry.

&& echo "OK: pulled digest matches the registry ($pulled)" \
|| { echo "FAIL: pulled=$pulled registry=$registry" >&2; exit 1; }
```

**PASS**: the printed digest equals the Docker Hub digest from step 2.
**PASS**: `OK`.

## 6. The image contains the released code

Expand Down Expand Up @@ -166,14 +208,53 @@ The chart and compose pins are part of the release surface; shipping them
pointing at the previous version is a silent regression for anyone deploying
from the tag.

Compare against `$VER` exactly. A filter that merely matches something
version-shaped is satisfied by a stale pin left at the previous release — which
is the regression this step is meant to catch.

```bash
fail=0
# appVersion must be $VER without its leading v
want_app=${VER#v}
got_app=$(git show "$VER:infra/charts/openrag-stack/Chart.yaml" \
| awk -F'"' '/^appVersion:/{print $2}')
[ "$got_app" = "$want_app" ] \
&& echo "OK appVersion=$got_app" \
|| { echo "FAIL appVersion=$got_app want=$want_app"; fail=1; }

# Each OpenRag image in the chart, checked by repository. Do NOT just count
# version-shaped tags: values.yaml also pins third-party images (vllm, milvus,
# infinity) whose versions have nothing to do with this release.
# An empty result means the values layout changed and this check no longer finds
# the pin — that is a FAIL, not a pass.
for repo in 'linagora/openrag-ray' 'linagoraai/openrag-admin-ui' 'linagoraai/openrag'; do
got=$(git show "$VER:infra/charts/openrag-stack/values.yaml" \
| grep -A4 "repository: \"$repo\"$" \
| awk -F'"' '/^[[:space:]]*tag:/{print $2; exit}')
[ "$got" = "$VER" ] \
&& echo "OK $repo -> $got" \
|| { echo "FAIL $repo -> ${got:-NOT FOUND} (want $VER)"; fail=1; }
done

# compose pins (2 expected: openrag, openrag-admin-ui)
cpins=$(git show "$VER:infra/compose/docker-compose.yaml" \
| grep -cE "image: linagoraai/openrag(-admin-ui)?:$VER$")
[ "$cpins" -eq 2 ] \
&& echo "OK 2 compose pins at $VER" \
|| { echo "FAIL $cpins compose pins at $VER (expected 2)"; fail=1; }
Comment on lines +239 to +244

@coderabbitai coderabbitai Bot Jul 31, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Validate each compose image exactly.

$VER is interpolated into an extended regular expression. For v2.1.0, each . matches any character, so a value such as v2x1y0 can pass.

The count also does not require one openrag pin and one openrag-admin-ui pin. Duplicate pins can pass, and an additional stale pin is ignored. The unanchored pattern can also count commented lines.

Use fixed-string or field-based matching. Require exactly one active pin for each expected repository, and reject extra relevant pins.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/RELEASING.md around lines 239 - 244, Update the compose pin
validation in the “compose pins” check to avoid interpolating VER into an
unescaped regular expression and to ignore commented lines. Validate active
image fields using fixed-string or field-based matching, requiring exactly one
linagoraai/openrag:$VER pin and one linagoraai/openrag-admin-ui:$VER pin, while
failing when any additional relevant compose image pins exist.


[ "$fail" -eq 0 ] && echo "step 8 PASS" || { echo "step 8 FAIL" >&2; exit 1; }
```

Chart `version` is bumped independently of `appVersion` (it tracks chart
changes, not the app release), so check it by eye against the previous release
rather than against `$VER`:

```bash
git show "$VER:infra/charts/openrag-stack/Chart.yaml" | grep -E '^(version|appVersion)'
git show "$VER:infra/charts/openrag-stack/values.yaml" | grep -nE 'tag: "v[0-9]'
git show "$VER:infra/compose/docker-compose.yaml" | grep -nE 'image: linagoraai/'
git show "$VER:infra/charts/openrag-stack/Chart.yaml" | grep -E '^version:'
```

**PASS**: every OpenRag image pin reads `$VER`, `appVersion` matches, chart
`version` was bumped.
**PASS**: `step 8 PASS`, and chart `version` moved.

---

Expand Down
Loading