Skip to content

Commit d90bf11

Browse files
[CRE] mixed-env nightly: Slack-alert #cre-platform-operations on failure
A scheduled sweep has no PR to block, so a failure was only visible via GitHub's default single-recipient email + the Actions tab. Add a notify-on-failure job (if: failure(), needs the sweep) that posts a rotating-light alert to the #cre-platform-operations Slack channel with a link to the run, mirroring the existing CRE Slack-notify pattern (slack-github-action + QA_SLACK_API_KEY). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent abd8e65 commit d90bf11

2 files changed

Lines changed: 46 additions & 1 deletion

File tree

.github/workflows/cre-mixed-env-nightly.yaml

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,3 +58,48 @@ jobs:
5858
# develop builds and surfaces non-determinism introduced since then.
5959
mixed_env_baseline_image: "${{ secrets.QA_AWS_ACCOUNT_NUMBER }}.dkr.ecr.${{ secrets.QA_AWS_REGION }}.amazonaws.com/chainlink:nightly-${{ needs.set-dates.outputs.baseline }}"
6060
full_matrix: true
61+
62+
# There is no PR to block on a scheduled run, so alert the CRE team on Slack when
63+
# the sweep fails (a failure usually means non-determinism was introduced into
64+
# develop between the two nightlies, or a job/infra error).
65+
notify-on-failure:
66+
name: Notify Slack on failure
67+
if: failure()
68+
needs: [set-dates, mixed-env-sweep]
69+
runs-on: ubuntu-latest
70+
environment:
71+
name: integration
72+
deployment: false
73+
steps:
74+
- name: Send Slack notification (#cre-platform-operations)
75+
uses: slackapi/slack-github-action@485a9d42d3a73031f12ec201c457e2162c45d02d # v2.0.0
76+
with:
77+
errors: "true"
78+
method: chat.postMessage
79+
token: ${{ secrets.QA_SLACK_API_KEY }}
80+
# If delivery fails, invite the bot to the channel, or replace the name
81+
# with the channel ID (Cxxxxxxxx) which chat.postMessage resolves more reliably.
82+
payload: |
83+
{
84+
"channel": "#cre-platform-operations",
85+
"text": "CRE Mixed-Env Nightly Sweep failed — possible non-determinism introduced into develop.",
86+
"blocks": [
87+
{
88+
"type": "section",
89+
"text": {
90+
"type": "mrkdwn",
91+
"text": "*:rotating_light: CRE Mixed-Env Nightly Sweep failed :rotating_light:*"
92+
}
93+
},
94+
{
95+
"type": "section",
96+
"text": {
97+
"type": "mrkdwn",
98+
"text": "Today's develop nightly `nightly-${{ needs.set-dates.outputs.today }}` vs baseline `nightly-${{ needs.set-dates.outputs.baseline }}`. A failure usually means *non-determinism was introduced into develop* between these nightlies (or a job/infra error). Run: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|${{ github.run_id }}>. Please investigate."
99+
}
100+
},
101+
{
102+
"type": "divider"
103+
}
104+
]
105+
}

core/scripts/cre/environment/docs/mixed-env.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ These lines are **silent** when every node runs the same code, so a single occur
7272
## When it runs
7373

7474
- **CI:** automatically on CRE-affecting PRs, in its **own** workflow `.github/workflows/cre-mixed-env-tests.yaml` (kept separate from `cre-system-tests.yaml` so that file stays simple). It runs the OCR3/DON2DON-heavy tests — `Test_CRE_V2_Suite_Bucket_A`, `Test_CRE_V2_Suite_Bucket_B`, and the `Test_CRE_V2_EVM_Read_*` suite — under mixed-env. The PR image and the develop image are both already built (per-PR and nightly), so no extra image builds are added. A dedicated **Check for non-determinism** step scans the live node containers after the suite and fails the job on any marker — kept separate from the auto-quarantined test step so the failure can't be swallowed.
75-
- **Nightly full sweep:** `.github/workflows/cre-mixed-env-nightly.yaml` calls the same dedicated workflow with `full_matrix: true` — running the *entire* matrix under mixed-env (EVM base + Solana/Aptos/Stellar/sharded/cache-test/multi-gateway variants), comparing today's develop nightly against a previous one to catch non-determinism introduced into develop between nightlies.
75+
- **Nightly full sweep:** `.github/workflows/cre-mixed-env-nightly.yaml` calls the same dedicated workflow with `full_matrix: true` — running the *entire* matrix under mixed-env (EVM base + Solana/Aptos/Stellar/sharded/cache-test/multi-gateway variants), comparing today's develop nightly against a previous one to catch non-determinism introduced into develop between nightlies. A scheduled run has no PR to block, so on failure it posts a `:rotating_light:` alert to the **#cre-platform-operations** Slack channel (with a link to the run) — that's the signal to watch, not a CI check.
7676

7777
## Required check & emergency bypass
7878

0 commit comments

Comments
 (0)