Skip to content

ci: make the regression harnesses report failures and not hang - #18

Open
doha-230 wants to merge 1 commit into
akhuettel:masterfrom
doha-230:pr/regression-harness-status
Open

doha-230 wants to merge 1 commit into
akhuettel:masterfrom
doha-230:pr/regression-harness-status

Conversation

@doha-230

Copy link
Copy Markdown

Both harnesses could report a broken build as a success, and one of them could hang until the CI job timeout:

  • build_and_run_all_cli_tests: the status of ../bin/$t was lost in "../bin/$t | sed ... | grep -v 'Using QtTest'": with "set -e" the status of a pipeline is the status of its last command, so a test binary that failed was reported as a success as long as grep printed a line. The status is now read before the coloring pipeline, every binary is bounded by ENGAUGE_CLI_TEST_TIMEOUT (default 300s), and the script exits non-zero when any test failed or timed out.
  • build_and_run_all_gui_tests: a FAIL was only printed, and the script exited 0 in every case, so a GUI regression could never fail the build. It now counts the failures and timeouts, prints a summary, and exits non-zero. Every case is bounded by ENGAUGE_GUI_TEST_TIMEOUT (default 120s), because a hanging Engauge process otherwise consumes the whole CI job. A hang is reported as TIMEOUT with the case name instead.
  • build_and_run_all_gui_tests: cases that download an image from a web site, such as drag_and_drop_http, depend on an external site and cannot run without internet access. ENGAUGE_SKIP_NETWORK_TESTS=1 skips them (opt-in; the default still runs every case).
  • ci-test.yml: qt6-image-formats-plugins was missing, so GIF/JPEG/TIFF loading fails. Engauge then shows a modal warning dialog and waits for input forever, which is what made the drag_and_drop regression cases hang.

Verified with a stub Engauge and a stub test binary: a failing binary now gives "FAIL : (exit status 1)" and exit status 1 (the old script printed PASS and exited 0), a hanging binary gives "TIMEOUT: " and exit status 1 instead of blocking, a passing binary still exits 0, and ENGAUGE_SKIP_NETWORK_TESTS=1 skips only the http cases.

Both harnesses could report a broken build as a success, and one of them could
hang until the CI job timeout:

- build_and_run_all_cli_tests: the status of ../bin/$t was lost in
  "../bin/$t | sed ... | grep -v 'Using QtTest'": with "set -e" the status of a
  pipeline is the status of its last command, so a test binary that failed was
  reported as a success as long as grep printed a line. The status is now read
  before the coloring pipeline, every binary is bounded by
  ENGAUGE_CLI_TEST_TIMEOUT (default 300s), and the script exits non-zero when
  any test failed or timed out.
- build_and_run_all_gui_tests: a FAIL was only printed, and the script exited 0
  in every case, so a GUI regression could never fail the build. It now counts
  the failures and timeouts, prints a summary, and exits non-zero. Every case is
  bounded by ENGAUGE_GUI_TEST_TIMEOUT (default 120s), because a hanging Engauge
  process otherwise consumes the whole CI job. A hang is reported as TIMEOUT
  with the case name instead.
- build_and_run_all_gui_tests: cases that download an image from a web site,
  such as drag_and_drop_http, depend on an external site and cannot run without
  internet access. ENGAUGE_SKIP_NETWORK_TESTS=1 skips them (opt-in; the default
  still runs every case).
- ci-test.yml: qt6-image-formats-plugins was missing, so GIF/JPEG/TIFF loading
  fails. Engauge then shows a modal warning dialog and waits for input forever,
  which is what made the drag_and_drop regression cases hang.

Verified with a stub Engauge and a stub test binary: a failing binary now gives
"FAIL : <test> (exit status 1)" and exit status 1 (the old script printed PASS
and exited 0), a hanging binary gives "TIMEOUT: <test>" and exit status 1
instead of blocking, a passing binary still exits 0, and
ENGAUGE_SKIP_NETWORK_TESTS=1 skips only the http cases.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant