fix(release): make the release checklist able to fail - #840
Conversation
Four checks in .github/RELEASING.md looked like gates but could not fail, which is the exact defect the document was written to prevent. Found in review of #839. - dg() piped a failed registry lookup into sha256sum, which hashes empty input and returns sha256:e3b0c442... — a real-looking digest. Step 2 then reported a missing image as present, and two missing tags compared equal so steps 3 and 4 printed OK. It now captures the manifest first and returns non-zero on an empty or failed lookup. Step 4 gained -n guards: with the helper fixed, two missing tags are both empty and would still have compared equal. - The job-conclusion check only printed the count of non-success jobs and exited 0 regardless, so a release could continue past a skipped build. It now exits non-zero. - Step 5 compared RepoDigests (repo@sha256:...) against a bare sha256:..., which can never match literally. It now strips the repository prefix. - Step 8 matched any version-shaped tag rather than comparing to $VER, so a pin left at the previous release satisfied it. It now checks each OpenRag repository's tag by name. Counting version-shaped tags instead would have been wrong: values.yaml also pins vllm, milvus and infinity, whose versions are unrelated to the release. Every snippet was executed rather than reasoned about: the job gate was run against the v2.0.1 run that built nothing (correctly fails) and against the v2.1.0 run (passes), and the rest against the live v2.1.0 registries.
📝 WalkthroughWalkthroughThe release checklist now rejects missing or empty digests, failed jobs, registry and image mismatches, and missing or incorrect chart, image-tag, and compose version pins. ChangesRelease validation
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
coderabbitai
Bot
added
the
chore
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/RELEASING.md:
- Around line 239-244: Update the compose pin validation in the “compose pins”
check to avoid interpolating VER into an unescaped regular expression and to
ignore commented lines. Validate active image fields using fixed-string or
field-based matching, requiring exactly one linagoraai/openrag:$VER pin and one
linagoraai/openrag-admin-ui:$VER pin, while failing when any additional relevant
compose image pins exist.
- Around line 82-88: Update the release-job validation around the bad count to
first verify that verify-tag, build-and-push-image, build-and-push-image-ray,
and build-and-push-image-admin-ui are all present in the gh run view result.
Fail the gate when any required job is missing, then retain the existing
non-success conclusion check before continuing.
- Around line 161-165: Update the Docker verification flow in
.github/RELEASING.md to fail immediately when docker pull fails, preventing
inspection of a stale cached image. When extracting pulled, select the
RepoDigests entry for linagoraai/openrag rather than indexing the unordered
collection, reject an empty digest, then compare that digest with registry.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: c5ac40e6-828f-4166-8744-f84e9563fdb0
📒 Files selected for processing (1)
.github/RELEASING.md
| bad=$(gh run view "$RUN_ID" --json jobs \ | ||
| --jq '[.jobs[] | select(.conclusion != "success")] | length') || exit 1 | ||
| if [ "$bad" -ne 0 ]; then | ||
| echo "FAIL: $bad job(s) did not conclude success — do not continue" >&2 | ||
| exit 1 | ||
| fi | ||
| echo "OK: every job concluded success" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Require all four release jobs before continuing.
This gate checks only the conclusions of returned jobs. If a required job is absent, bad is still 0 when the remaining jobs succeed. Assert that verify-tag, build-and-push-image, build-and-push-image-ray, and build-and-push-image-admin-ui are present before checking their conclusions.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/RELEASING.md around lines 82 - 88, Update the release-job validation
around the bad count to first verify that verify-tag, build-and-push-image,
build-and-push-image-ray, and build-and-push-image-admin-ui are all present in
the gh run view result. Fail the gate when any required job is missing, then
retain the existing non-success conclusion check before continuing.
| docker pull "linagoraai/openrag:$VER" | ||
| docker image inspect "linagoraai/openrag:$VER" --format '{{index .RepoDigests 0}}' | ||
| pulled=$(docker image inspect "linagoraai/openrag:$VER" \ | ||
| --format '{{index .RepoDigests 0}}' | cut -d@ -f2) | ||
| registry=$(dg "linagoraai/openrag:$VER") || { echo "FAIL: tag not in registry" >&2; exit 1; } | ||
| [ "$pulled" = "$registry" ] \ |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== file excerpt =="
sed -n '140,175p' .github/RELEASING.md 2>/dev/null || true
echo "== nearby command context =="
git show HEAD:.github/RELEASING.md 2>/dev/null | sed -n '140,175p' || true
echo "== repo docker/digest refs =="
rg -n "docker (pull|image inspect)|RepoDigests|linagoraai/openrag|registry|verify" .github/RELEASING.md README.md 2>/dev/null || trueRepository: linagora/openrag
Length of output: 5259
🌐 Web query:
Docker CLI image inspect RepoDigests multiple entries index first entry repository may not match pulled image documentation
💡 Result:
The RepoDigests field in Docker image metadata is a list of content-addressable digests of image manifests from which a local image was referenced [1]. Because this field is a collection of all registry/repository locations from which an image has been pulled or pushed, it can contain multiple entries if the same image ID exists under different repository names or registry paths [2][3]. It is important to understand the following regarding RepoDigests: 1. Order and Indexing: The order of entries in the RepoDigests list is not guaranteed to align with the specific image tag or repository used during a recent pull [2]. Attempting to retrieve a specific entry using index 0 (e.g., {{index.RepoDigests 0}}) is unreliable if multiple digests are present, as the first entry may not correspond to the image name you used to pull the image [2]. 2. Nature of the Field: RepoDigests are attributes of where an image is stored rather than immutable properties of the image itself [3]. They record the source manifest digest at the time of the pull operation [3]. Consequently, the repository portion of the digest string is derived from the registry/repository path used during that pull and may not match the repository name you are currently working with if the image was pulled from or pushed to multiple locations [2][3]. 3. deduplication: Docker has implemented fixes in recent versions to prevent duplicate entries from appearing in the RepoDigests list, ensuring that each unique reference is stored only once [4][5][6]. To reliably work with these values, avoid relying on positional indexing. If you need a specific digest for a particular repository, you should filter or iterate through the list using external tools (such as grep or platform-specific template logic) to match the desired registry path string, rather than assuming the first element is the correct one [2].
Citations:
- 1: https://github.com/moby/moby/blob/d74daf1a/api/types/image/image_inspect.go
- 2: https://stackoverflow.com/questions/32046334/where-can-i-find-the-sha256-code-of-a-docker-image
- 3: skopeo inspect digest differs from docker images --digest podman-container-tools/skopeo#469
- 4: c8d/inspect: Fix duplicate RepoDigests moby/moby#48777
- 5: [27.x] c8d/inspect: Fix duplicate RepoDigests moby/moby#48785
- 6: [backport 24.0] c8d/inspect: Don't duplicate digested ref moby/moby#46014
🌐 Web query:
Docker CLI image pull returns non-zero when pull fails separate from image inspect caching documented
💡 Result:
The Docker CLI docker pull command is designed to return a non-zero exit code when a pull operation fails [1]. This behavior is generally consistent with command-line standards where failures are signaled through exit status [2][3]. While some legacy reports or specific environmental configurations (such as certain uses of docker-compose) have occasionally raised questions about suppression of error codes [1], the docker pull command itself executes via the Docker daemon, which propagates errors returned during the pull process [4]. If the daemon encounters an error—such as an image not found, an authentication failure, or a network issue—the CLI client receives and reports this error, typically resulting in a non-zero exit status [5][4][1]. It is important to note that the Docker CLI's error handling and exit codes are not always granularly documented for every failure scenario [6]. In practice: - A failed docker pull due to registry unavailability, missing images, or access issues will generally result in a non-zero exit code [5][1]. - Some users have historically requested specific or distinct exit codes for different types of failures (e.g., distinguishing between authentication issues and network issues) [5][2], but these are not currently implemented as a standardized, stable API, and users should rely on the standard non-zero exit convention to detect failure [2]. - Behaviors may occasionally appear inconsistent across different versions or when used in combination with tools like docker-compose, which may have its own logic for handling or suppressing exit statuses [1]. There is no documented "caching" mechanism that would cause a failed pull to return a zero exit code under normal circumstances; if an image is already cached locally (pulled successfully previously), docker pull will verify its existence and return zero, which is the expected "success" behavior. If you are experiencing unexpected zero exit codes during a failed pull, ensure you are not masking the error in a shell script (e.g., using || true or checking the exit code of the wrong process) [2].
Citations:
- 1: https://forums.docker.com/t/why-docker-compose-pull-returns-zero-status-code-if-no-image-is-found-on-server/108427
- 2: [Proposal] docker pull return code docker/cli#538
- 3: Docker commands should return nonzero error codes on failure moby/moby#354
- 4: https://github.com/moby/docker/blob/v28.5.2/daemon/images/image_pull.go
- 5: docker-pull return codes moby/moby#27658
- 6: docker pull is now interactive moby/moby#50204
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
text = Path(".github/RELEASING.md").read_text()
lines = text.splitlines()
print("contains ignored pull:", "docker pull" in text and "||" in "\n".join(lines[158:164]))
print("line 161-165:")
for i in range(157, min(166, len(lines))):
print(f"{i+1}: {lines[i]}")
PY
if command -v docker >/dev/null 2>&1; then
echo "== local Docker availability =="
docker version 2>&1 | sed -n '1,20p'
else
echo "== local Docker availability: not installed =="
fiRepository: linagora/openrag
Length of output: 661
Fail on Docker pull errors and compare only the matching repository digest.
docker pull status is ignored, so the script can inspect an older locally cached image after a failed pull and may still pass if that stale digest equals the registry value. Also avoid {{index .RepoDigests 0}} because RepoDigests is an unordered collection. Use the pull exit status, select only the linagoraai/openrag entry, reject an empty digest, then compare with registry.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/RELEASING.md around lines 161 - 165, Update the Docker verification
flow in .github/RELEASING.md to fail immediately when docker pull fails,
preventing inspection of a stale cached image. When extracting pulled, select
the RepoDigests entry for linagoraai/openrag rather than indexing the unordered
collection, reject an empty digest, then compare that digest with registry.
| # compose pins (2 expected: openrag, openrag-admin-ui) | ||
| cpins=$(git show "$VER:infra/compose/docker-compose.yaml" \ | ||
| | grep -cE "image: linagoraai/openrag(-admin-ui)?:$VER$") | ||
| [ "$cpins" -eq 2 ] \ | ||
| && echo "OK 2 compose pins at $VER" \ | ||
| || { echo "FAIL $cpins compose pins at $VER (expected 2)"; fail=1; } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Validate each compose image exactly.
$VER is interpolated into an extended regular expression. For v2.1.0, each . matches any character, so a value such as v2x1y0 can pass.
The count also does not require one openrag pin and one openrag-admin-ui pin. Duplicate pins can pass, and an additional stale pin is ignored. The unanchored pattern can also count commented lines.
Use fixed-string or field-based matching. Require exactly one active pin for each expected repository, and reject extra relevant pins.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/RELEASING.md around lines 239 - 244, Update the compose pin
validation in the “compose pins” check to avoid interpolating VER into an
unescaped regular expression and to ignore commented lines. Validate active
image fields using fixed-string or field-based matching, requiring exactly one
linagoraai/openrag:$VER pin and one linagoraai/openrag-admin-ui:$VER pin, while
failing when any additional relevant compose image pins exist.
1 participant
Fixes the four issues CodeRabbit raised on
.github/RELEASING.mdin #839. All four were valid, and they share a theme: the checks looked like gates but could not fail — the exact defect the document was written to prevent.Raised there rather than fixed there because #839 was a pure back-merge, and
developdismisses stale reviews on push.What was wrong
dg()sha256sum, which hashes empty input and returnssha256:e3b0c442…— a real-looking digest. Step 2 reported a missing image as present, and two missing tags compared equal so steps 3 and 4 printedOK.RepoDigests(repo@sha256:…) against a baresha256:…. Those can never match literally.$VER, so a pin left at the previous release satisfied it.What changed
dg()captures the manifest first and returns non-zero on an empty or failed lookup. Step 4 also gained-nguards: with the helper fixed, two missing tags are both empty and would still have compared equal.values.yamlalso pins vllm, milvus and infinity, whose versions are unrelated to the release. My first attempt did exactly that and flaggedvllm/vllm-openai:v0.11.2as a stale OpenRag pin.Verification
Every snippet was executed, not reasoned about:
30034799802) — correctly fails with3 job(s) did not conclude success— and against the v2.1.0 run (30620163814) — passes.dg()on a nonexistent tag now returns nothing and non-zero; two missing tags no longer compare equal.v2.1.0tag:step 8 PASS, exit 0.No change to the v2.1.0 release, which was verified with real, distinct digests and independently confirmed by pulling the image and reading
2.1.0out of it.Summary by CodeRabbit