Skip to content

[rhoai-2.25] chore(ci): RPM lock renewal workflow + CI triage docs for rhoai-2.25 - #2643

Merged
jiridanek merged 3 commits into
rhoai-2.25from
fix/rhoai-2.25-rpms-lock-renewal-workflow
Jul 30, 2026
Merged

[rhoai-2.25] chore(ci): RPM lock renewal workflow + CI triage docs for rhoai-2.25#2643
jiridanek merged 3 commits into
rhoai-2.25from
fix/rhoai-2.25-rpms-lock-renewal-workflow

Conversation

@jiridanek

@jiridanek jiridanek commented Jul 30, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds .github/workflows/rpms-lock-renewal.yaml (manual workflow_dispatch only) so RHDS RPM locks can be refreshed without combining with pylock / piplock-renewal.
  • Defaults: variant=rhds, branch=rhoai-2.25; subscription/git-crypt are optional (public UBI path on this branch).
  • Adds docs/ci-failure-triage.md explaining hermetic dnf NEVR conflicts (cannot install both … @System) as stale Hermeto pins vs newer base image, with fix = run this Action.
  • Points Agents.md at that triage doc.

Companion lock refresh: #2642

Test plan

  • Workflow appears under Actions → RPM Lock Files Renewal Action
  • Dispatch with variant=rhds, branch=rhoai-2.25 (after merge) opens/updates an RPM-only PR
  • Confirm doc links from Agents.md resolve

Made with Cursor

Summary by CodeRabbit

  • Documentation
    • Added a new CI failure triage guide for hermetic build issues on the relevant branch, including a step-by-step runbook for outdated RPM lock pins.
    • Updated Agents documentation to link to the new triage page and expanded “Common Issues” with a dedicated RPM/lock conflict item.
  • CI / Automation
    • Improved RPM lockfile renewal workflow messaging and clarified branch-specific behavior differences.
    • Made registry auth setup conditional on having a valid pull-secret JSON; updated the fallback message when unavailable or invalid.

Backport a manual-only rpms-lock-renewal Action for rhoai-2.25 (separate from
pylock), and document that hermetic dnf NEVR conflicts usually mean stale
Hermeto RPM pins that should be relocked via that workflow.

Co-authored-by: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: db9b5a37-6dc7-4bf3-b47e-9d152c44de56

📥 Commits

Reviewing files that changed from the base of the PR and between e861502 and c2836ae.

📒 Files selected for processing (1)
  • .github/workflows/rpms-lock-renewal.yaml

📝 Walkthrough

Walkthrough

Changes

CI failure triage

Layer / File(s) Summary
RPM lock failure runbook and contributor guidance
Agents.md, docs/ci-failure-triage.md
Adds rhoai-2.25 guidance for Hermeto/DNF NEVR conflicts, RPM lock regeneration through workflow dispatch or locally, related failure pointers, and troubleshooting references.
Workflow branch context and registry authentication
.github/workflows/rpms-lock-renewal.yaml
Documents public UBI repository usage for rhoai-2.25 and installs registry authentication only when the pull-secret is a valid JSON object.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested labels: lgtm, approved

Suggested reviewers: atheo89, daniellutz

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the RPM lock renewal workflow and CI triage doc updates for rhoai-2.25.
Description check ✅ Passed The description includes a solid summary and test plan for the changes, with only the template checklist sections missing.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/rhoai-2.25-rpms-lock-renewal-workflow

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Superseded by newer run: https://github.com/red-hat-data-services/notebooks/actions/runs/30527933835

CI status [antigravity]

Run: Build Notebooks (push) #305279290242/2 complete · 2 skipped
Last updated: 2026-07-30T08:45:34Z

Workflow completed with skipped jobs.

@jiridanek

Copy link
Copy Markdown
Member Author

Important difference from main

On rhoai-2.25, this workflow does not require a RHEL subscription, unlike the RHDS path on main.

Reason:

  • The only image that RPM-prefetches / ships Hermeto rpms.lock.yaml on this branch is codeserver.
  • Codeserver here is built from UBI (ubi9/python-312) and resolves RPMs from public UBI repos.
  • There is no other RPM-prefetch consumer on rhoai-2.25 that needs entitled RHEL content.

So for dispatch on this branch: variant=rhds, branch=rhoai-2.25, and leave subscription / git-crypt secrets unset — lock regen should use public UBI only.

On main, RHDS lock renewal can still need subscription secrets because the broader hermetic / entitled RPM set is different.

Codeserver is the only Hermeto RPM consumer on this branch and runs on public
UBI, unlike main's RHDS path that may still require subscription secrets.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Superseded by newer run: https://github.com/red-hat-data-services/notebooks/actions/runs/30528028508

CI status [antigravity]

Run: Build Notebooks (pr) #305279338353/3 complete · 1 passed · 2 cancelled
Last updated: 2026-07-30T08:47:10Z after workflow update completed

Failures so far

Job Failed step Link
codeserver-ubi9-python-3.12 · linux/arm64 [rhoai] / build Run actions/checkout@3d3c42e logs
codeserver-ubi9-python-3.12 · linux/amd64 [rhoai] / build Run actions/checkout@3d3c42e logs

Likely root causes

  • The build jobs codeserver-ubi9-python-3.12 · linux/arm64 [rhoai] / build and codeserver-ubi9-python-3.12 · linux/amd64 [rhoai] / build were cancelled during the checkout step (Run actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1) while fetching submodules (specifically cloning codeserver/ubi9-python-3.12/prefetch-input/code-server/lib/vscode).

Suggested next steps

  • Re-run the cancelled workflow jobs to determine if the submodule fetch/checkout failure was transient or due to external timeout/cancellation.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/rpms-lock-renewal.yaml:
- Around line 85-93: Update the “Configure registry auth from pull-secret” step
to copy pull-secret.json only when it is successfully unlocked or passes JSON
validation; otherwise leave Podman auth unchanged and continue with the public
UBI repositories path. Replace the existence-only condition while preserving the
existing destination and fallback message.

In `@docs/ci-failure-triage.md`:
- Around line 57-63: Update the “Local equivalent” example to cover both
lockfile locations renewed by the workflow, including the
prefetch-input/rhds/rpms.in.yaml input; alternatively, clearly label the
existing command as codeserver-only.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0483467a-3df0-4dfb-96d9-70bfd3da379a

📥 Commits

Reviewing files that changed from the base of the PR and between 41d6bb1 and 4f23566.

📒 Files selected for processing (3)
  • .github/workflows/rpms-lock-renewal.yaml
  • Agents.md
  • docs/ci-failure-triage.md

Comment thread .github/workflows/rpms-lock-renewal.yaml
Comment thread docs/ci-failure-triage.md
Avoid copying a git-crypt ciphertext blob into Podman auth.json when
GIT_CRYPT_KEY is absent on the public-UBI rhoai-2.25 path.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jiridanek

Copy link
Copy Markdown
Member Author

Addressed the CodeRabbit finding: pull-secret.json is only installed as Podman auth.json when jq confirms it is a JSON object (unlocked/plaintext). Encrypted git-crypt blobs fall through to the public-UBI path.

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Superseded by newer run: https://github.com/red-hat-data-services/notebooks/actions/runs/30528411559

CI status [antigravity]

Run: Build Notebooks (pr) #305280285083/3 complete · 1 passed · 1 failed · 1 cancelled
Last updated: 2026-07-30T08:53:16Z after workflow update completed

Failures so far

Job Failed step Link
codeserver-ubi9-python-3.12 · linux/amd64 [rhoai] / build Install from uv.lock (project + dev deps) logs
codeserver-ubi9-python-3.12 · linux/arm64 [rhoai] / build Build: make codeserver-ubi9-python-3.12 logs

Likely root causes

  • Hermetic DNF package conflict (RPM lock pins out of date): The codeserver-ubi9-python-3.12 · linux/arm64 [rhoai] / build job failed during dnf install due to a NEVR conflict (cannot install both nodejs-1:22.23.1-1.module+... and nodejs-1:22.23.1-2.module+... from @System), which indicates that the Hermeto/RPM lock pins are stale relative to the newer floating base image.
  • Job cancellation: The codeserver-ubi9-python-3.12 · linux/amd64 [rhoai] / build job was cancelled (likely due to workflow fail-fast behavior or failure of the parallel arm64 job).

Suggested next steps

  • Regenerate the RPM lockfiles (rpms.lock.yaml) using the newly added RPM Lock Files Renewal Action (.github/workflows/rpms-lock-renewal.yaml) as outlined in docs/ci-failure-triage.md.
  • Re-run the PR workflow after updating the lock pins to resolve the dnf package conflicts.

@github-actions

Copy link
Copy Markdown

CI status [antigravity]

Run: Build Notebooks (pr) #305284115593/3 complete · 1 passed · 2 failed
Last updated: 2026-07-30T08:59:10Z after workflow update completed

Failures so far

Job Failed step Link
codeserver-ubi9-python-3.12 · linux/arm64 [rhoai] / build Build: make codeserver-ubi9-python-3.12 logs
codeserver-ubi9-python-3.12 · linux/amd64 [rhoai] / build Build: make codeserver-ubi9-python-3.12 logs

Likely root causes

  • Hermetic dnf install failures during codeserver builds (codeserver-ubi9-python-3.12 · linux/amd64 [rhoai] / build and codeserver-ubi9-python-3.12 · linux/arm64 [rhoai] / build) caused by package version conflicts (e.g., cannot install both nodejs-1:22.23.1-1.module+... and nodejs-1:22.23.1-2.module+... from @System), which typically indicate that Hermeto RPM lock pins (rpms.lock.yaml) are out of date relative to the floating UBI base image.
  • A secondary python execution error (ModuleNotFoundError: No module named 'ci.logging_config') encountered when running /home/runner/work/notebooks/notebooks/scripts/monitor_resources.py.

Suggested next steps

  • Regenerate and update the RPM lock files using the newly added docs/ci-failure-triage.md playbook and the RPM Lock Files Renewal Action workflow (.github/workflows/rpms-lock-renewal.yaml).
  • Verify Python path or backport ci/logging_config.py as noted in the CI failure triage documentation to resolve resource monitoring script errors.

@ide-developer ide-developer left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good — clean separation of RPM lock renewal from pylock/piplock, solid docs.

Workflow (.github/workflows/rpms-lock-renewal.yaml)

  • Checkout pin (actions/checkout@…v7.0.1) and login-action pin match the existing piplock-renewal workflow — consistent.
  • Good security hygiene: persist-credentials: false + PAT push, secrets via env: not string interpolation, ::add-mask:: for subscription creds.
  • permissions: {} at workflow level + scoped per-job is the recommended pattern.
  • The RPM_INPUTS array handles both the top-level prefetch-input/${VARIANT}/rpms.in.yaml (exists on main but not on rhoai-2.25) and the codeserver path — the [[ ! -f … ]] skip logic is correct.
  • Nit: the if: github.event_name == 'workflow_dispatch' guard on the job is redundant since workflow_dispatch is the only trigger. Harmless but could be dropped for cleanliness.

Docs (docs/ci-failure-triage.md)

  • Well-structured symptom → cause → fix pattern. The explanation of why hermetic dnf NEVR conflicts happen (locked pins vs floating base image) is clear and actionable.
  • The "other failure classes" table is a nice bonus for common issues.

Agents.md additions

  • Cross-references in three relevant spots (CI section, build failures, getting help). Good discoverability.

LGTM.

@openshift-ci

openshift-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ide-developer
Once this PR has been reviewed and has the lgtm label, please assign caponetto for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jiridanek
jiridanek merged commit bb1413c into rhoai-2.25 Jul 30, 2026
32 of 35 checks passed
@jiridanek
jiridanek deleted the fix/rhoai-2.25-rpms-lock-renewal-workflow branch July 30, 2026 09:06
@jiridanek jiridanek changed the title chore(ci): RPM lock renewal workflow + CI triage docs for rhoai-2.25 [rhoai-2.25] chore(ci): RPM lock renewal workflow + CI triage docs for rhoai-2.25 Jul 30, 2026
@jiridanek

Copy link
Copy Markdown
Member Author

Follow-up for the ubuntu-latest → 24.04 AppArmor failure after merge: same-repo fix + validation is in #2645 / run 30529943934 (Install Podman OK on ubuntu-26.04; auto-PR push still blocked by empty GH_ACCESS_TOKEN).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants