Add ATEX testing to the upstream CI workflows #14203

ggbecker · 2025-12-02T11:38:04Z

Description:

Add ATEX testing to the upstream CI workflows
It posts the resulting tests link as a comment to the pull request when it finishes.

Rationale:

This aims to replace existing testing farm individual checks to a centralized ATEX execution that can run all CI upstream tests for PRs and manage them accordingly.
The PR as of now, runs only a single STIG hardening test on Centos Stream 8/9/10 to be served as a proof of concept. We should eventually extend to include more tests similarly as the current upstream CI.

ggbecker · 2025-12-02T14:49:06Z

From: https://github.com/ComplianceAsCode/content/actions/runs/19858710726/job/56915038821?pr=14203

   File "/usr/local/lib/python3.13/site-packages/atex/provisioner/testingfarm/api.py", line 507, in reserve
    if self.api.whoami()["token"]["ranch"] == "public":
       ~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/atex/provisioner/testingfarm/api.py", line 122, in whoami
    raise ValueError("whoami() requires an auth token")
ValueError: whoami() requires an auth token

I'm afraid the token will only be allowed to be used when we merge the pull request

https://github.com/ComplianceAsCode/content/pull/14203/files#diff-9581118f6672453f95900179c8eccc554c1b111682dab06bd16317909fdaf295R109

The same code with the same token is working fine on my fork: ggbecker#41

tests/submit_results_to_testing_farm.py

tests/run_tests_testingfarm.py

.github/workflows/atex.yaml

tests/submit_results_to_testing_farm.py

Mab879 · 2025-12-02T19:56:34Z

tests/submit_results_to_testing_farm.py

+
+# do faster queries than the default 30 secs, because we don't track
+# many dozens of requests, just one
+class FastRequest(api.Request):


Consider adding a docstring for why this class is needed.

added, based on the existing test, coming from our productization ATEX scripts

Mab879 · 2025-12-02T20:39:03Z

.github/workflows/atex-test.yaml

+
+      - name: Run tests on Testing Farm
+        env:
+          TESTING_FARM_API_TOKEN: ${{ secrets.TESTING_FARM_API_TOKEN }}


IIRC, this will never be available since we are using pull_request.

We might need break this out into two jobs. One that builds using pull_request, then a second one that uses workflow_run to trigger the tests.

I have done this separation but gemini tells me this would only work when the file is already merged in the repo for example, that's why we don't see a check running for this workflow.

ggbecker · 2025-12-03T13:44:52Z

I've addressed all the feedback provided and split the jobs into two workflows, one with the workflow_run as suggested by @Mab879 . Let's see how it behaves.

.github/workflows/atex-build.yaml

.github/workflows/atex-test.yaml

.github/workflows/atex-build.yaml

.github/workflows/atex-test.yaml

comps · 2025-12-04T08:38:18Z

How would I make a change in the workflow (later down the road) while being able to test the change in a PR, and not hit any of the secrets limitations?

For example, if we switch to pip install atex==0.20 or so, and I want to upgrade to pip install atex==1.0, which brings an incompatible API change, how can I change the workflow code (to adjust for new API) while being able to test the change?

Will it Just Work?

Also nitpicks:

Contest is technically a "test suite", not a "framework" (though, arguably, Contest contains a framework inside lib/)
This PR is about running Contest in upstream PRs, not about running "Testing Farm", ... TF is one of the backends we can use to run Contest, but not a type of test / suite, so the bot comment is slightly confusing

ggbecker · 2025-12-04T14:23:22Z

How would I make a change in the workflow (later down the road) while being able to test the change in a PR, and not hit any of the secrets limitations?

For example, if we switch to pip install atex==0.20 or so, and I want to upgrade to pip install atex==1.0, which brings an incompatible API change, how can I change the workflow code (to adjust for new API) while being able to test the change?

Will it Just Work?

I believe it will not. Maybe if you push to a branch belonging to ComplianceAsCode the situation changes, at least that's what worked here #14209.

But then I don't know about the workflow_run type that may behave a bit differently. I'm going to investigate this better to see if I can find the relevant information.

ggbecker · 2025-12-05T11:23:18Z

/packit retest-failed

Also added restrictions to token permissions as suggested by github.

Use env in the shebang for python scripts. Use fixed hash for github actions to prevent supply chain attacks. Remove dead code. Add pcre2 to the requirements.txt file.

…e repo.

There's no need to manage issues from the ATEX PR point of view.

Mab879 · 2025-12-05T14:31:24Z

I believe it will not. Maybe if you push to a branch belonging to ComplianceAsCode the situation changes, at least that's what worked here #14209.

Sadly, this is a feature not bug. So that we don't expose our secrets.

openshift-ci · 2025-12-05T15:22:52Z

@ggbecker: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-openshift-node-compliance	`8b45e57`	link	true	`/test e2e-aws-openshift-node-compliance`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ggbecker added this to the 0.1.80 milestone Dec 2, 2025

ggbecker requested review from Mab879, jan-cerny, matusmarhefka and vojtapolasek as code owners December 2, 2025 11:38

ggbecker added the Test Suite Update in Test Suite. label Dec 2, 2025

ggbecker force-pushed the atex-workflow-tests-3 branch from f1dae23 to 301dab3 Compare December 2, 2025 12:32

Mab879 requested changes Dec 2, 2025

View reviewed changes

Mab879 reviewed Dec 2, 2025

View reviewed changes

ggbecker mentioned this pull request Dec 3, 2025

[duplicate] Add ATEX testing to the upstream CI workflows #14209

Open

Mab879 self-assigned this Dec 3, 2025

Mab879 reviewed Dec 3, 2025

View reviewed changes

ggbecker force-pushed the atex-workflow-tests-3 branch from ccbbfcb to 8d50245 Compare December 4, 2025 14:14

ggbecker added 11 commits December 5, 2025 14:59

Add new atex test workflow.

0335cfe

Refactor ATEX workflow code.

b0ea61e

Split workflows into separate jobs for better secret handling.

31f1b98

Also added restrictions to token permissions as suggested by github.

Apply suggestions.

52a610b

Use env in the shebang for python scripts. Use fixed hash for github actions to prevent supply chain attacks. Remove dead code. Add pcre2 to the requirements.txt file.

Refactor testingfarm scripts to follow pattern from other files in th…

85a41c7

…e repo.

Make sure the workflow run triggers from the right branches.

8ebcd20

Freeze all github actions to latest available version at this moment.

5d76326

Remove unnecessary permission from ATEX github workflow.

ec33062

There's no need to manage issues from the ATEX PR point of view.

Save the PR info only once.

a82cd95

Fix nitpicks.

6a807ce

Add docstring to the FastRequest class.

8b45e57

ggbecker force-pushed the atex-workflow-tests-3 branch from 0c53819 to 8b45e57 Compare December 5, 2025 14:00

Add ATEX testing to the upstream CI workflows #14203

Are you sure you want to change the base?

Add ATEX testing to the upstream CI workflows #14203

Conversation

ggbecker commented Dec 2, 2025

Description:

Rationale:

Uh oh!

ggbecker commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Mab879 Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

ggbecker Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

Mab879 Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

ggbecker Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

ggbecker commented Dec 3, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

comps commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggbecker commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggbecker commented Dec 5, 2025

Uh oh!

Mab879 commented Dec 5, 2025

Uh oh!

openshift-ci bot commented Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ggbecker commented Dec 2, 2025 •

edited

Loading

comps commented Dec 4, 2025 •

edited

Loading

ggbecker commented Dec 4, 2025 •

edited

Loading