Repository navigation
Conversation
Both harnesses could report a broken build as a success, and one of them could hang until the CI job timeout: - build_and_run_all_cli_tests: the status of ../bin/$t was lost in "../bin/$t | sed ... | grep -v 'Using QtTest'": with "set -e" the status of a pipeline is the status of its last command, so a test binary that failed was reported as a success as long as grep printed a line. The status is now read before the coloring pipeline, every binary is bounded by ENGAUGE_CLI_TEST_TIMEOUT (default 300s), and the script exits non-zero when any test failed or timed out. - build_and_run_all_gui_tests: a FAIL was only printed, and the script exited 0 in every case, so a GUI regression could never fail the build. It now counts the failures and timeouts, prints a summary, and exits non-zero. Every case is bounded by ENGAUGE_GUI_TEST_TIMEOUT (default 120s), because a hanging Engauge process otherwise consumes the whole CI job. A hang is reported as TIMEOUT with the case name instead. - build_and_run_all_gui_tests: cases that download an image from a web site, such as drag_and_drop_http, depend on an external site and cannot run without internet access. ENGAUGE_SKIP_NETWORK_TESTS=1 skips them (opt-in; the default still runs every case). - ci-test.yml: qt6-image-formats-plugins was missing, so GIF/JPEG/TIFF loading fails. Engauge then shows a modal warning dialog and waits for input forever, which is what made the drag_and_drop regression cases hang. Verified with a stub Engauge and a stub test binary: a failing binary now gives "FAIL : <test> (exit status 1)" and exit status 1 (the old script printed PASS and exited 0), a hanging binary gives "TIMEOUT: <test>" and exit status 1 instead of blocking, a passing binary still exits 0, and ENGAUGE_SKIP_NETWORK_TESTS=1 skips only the http cases.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both harnesses could report a broken build as a success, and one of them could hang until the CI job timeout:
Verified with a stub Engauge and a stub test binary: a failing binary now gives "FAIL : (exit status 1)" and exit status 1 (the old script printed PASS and exited 0), a hanging binary gives "TIMEOUT: " and exit status 1 instead of blocking, a passing binary still exits 0, and ENGAUGE_SKIP_NETWORK_TESTS=1 skips only the http cases.