Skip to content

Activate nightly workflow for random and stress tests - #3500

Merged
kdeldycke merged 1 commit into
pallets:mainfrom
kdeldycke:activate-random-and-stress-tests
May 23, 2026
Merged

Activate nightly workflow for random and stress tests#3500
kdeldycke merged 1 commit into
pallets:mainfrom
kdeldycke:activate-random-and-stress-tests

Conversation

@kdeldycke

@kdeldycke kdeldycke commented May 22, 2026

Copy link
Copy Markdown
Collaborator

In #3139 we added stress tests and in #3151 we added randomization and parallel tests.

We decided to not activate them by default in #3151 (comment) and rely on contributors diligence to run them by hand to catch regressions. In practice nobody does. I even often forget to run them on my own during my sessions, only focused on the task at hand.

I'd like to re-assess this decision in light of regressions that could have been caught if these test were active. See the following PRs:

Also why even support parallel tests if we don't exercice them regularly? (ref: #2899)

Currently these tests adds ~15 minutes run-time. If you find this too long and invasive, I propose a compromise and run them nightly. But my preference would be to run them as part of the standard CI to incentivize contributors. And reduce surprises and and follow-up fixes.

Given the reversal in policy this change imply, I'm aiming for the next 8.5.0 minor release. I'm opening this PR as a way to discuss the best strategy forward.

@kdeldycke kdeldycke added f:test runner feature: cli test runner github_actions Pull requests that update GitHub Actions code labels May 22, 2026
@kdeldycke kdeldycke added this to the 8.5.0 milestone May 22, 2026
@kdeldycke
kdeldycke marked this pull request as draft May 22, 2026 11:07
@kdeldycke
kdeldycke merged commit 5b1aba8 into pallets:main May 23, 2026
24 of 25 checks passed
@Rowlando13

Copy link
Copy Markdown
Member

@kdeldycke I agree with the nightly run.

@kdeldycke
kdeldycke deleted the activate-random-and-stress-tests branch May 27, 2026 07:48
@kdeldycke

Copy link
Copy Markdown
Collaborator Author

I think we were overly cautious just running these tests nightly as in the end they only take 5 minutes, not 15: https://github.com/pallets/click/actions/runs/26496766042/usage . To me the cost-benefit is excellent to trigger on every PR: a 5 minute wait is nothing in comparison to the effort and synchronization time required to fix leaky issues from non-core maintainers. But I will let that decision for later.

Thanks @Rowlando13 for the merge!

Rowlando13 pushed a commit that referenced this pull request Jun 4, 2026
@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Jul 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

f:test runner feature: cli test runner github_actions Pull requests that update GitHub Actions code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants