chore: Flaky test handling

[netopyr]

Description:

This PR adds automated flaky-test retry and reporting. This is the first version. A more sophisticated handling and reporting will come soon.

A failing test is automatically retried twice. If it succeeds on one of the retries, it is considered flaky and not a failure. A flaky test no longer fails a build.

The specific reporting depends on the workflow:

• Node PR Checks: When a flaky test is discovered for the first time, we add a comment to the PR. The PR owner has to report the flaky test as previously.
• Dry runs: On the summary page, there is a new section Report Flaky Tests that will show any flaky tests. The person who kicked off the dry run should check if a new flaky was detected and report it. This is admittedly the weakest solution that will be improved soon.
• MATS runs on main: If the run fails for another reason, the error messages contain a new section for flaky tests. If the run succeeds but contains flaky tests, we automatically report a warning.
• XTS runs on main: The report contains a new section for the flaky tests.

[trunk-io]

Merging to main in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

[lfdt-bot]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scanner	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

@@            Coverage Diff            @@
##               main   #24031   +/-   ##
=========================================
  Coverage     78.29%   78.29%           
  Complexity    11658    11658           
=========================================
  Files          2461     2461           
  Lines         92858    92858           
  Branches       9945     9945           
=========================================
  Hits          72700    72700           
+ Misses        16489    16479   -10     
- Partials       3669     3679   +10     

see 14 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[codacy-production]

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ +0.01% (target: -1.00%)	✅ ∅

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (74f6aa7)	92761	76320	82.28%
Head commit (5ba35dd)	92761 (+0)	76330 (+10)	82.29% (+0.01%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#24031)	0	0	∅ (not applicable)

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences