Quarantined Tests #3152
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Executes quarantined tests | |
| # | |
| # COPILOT INSTRUCTIONS: | |
| # - Keep the shared 'paths:' entries (specialized-test-runner.yml, | |
| # run-tests.yml, build-cli-e2e-image.yml) in sync across | |
| # tests-outerloop.yml and tests-quarantine.yml. Each workflow also lists itself. | |
| # - Validate that each path exists in the repository before adding or | |
| # updating the list | |
| # - No external YAML file is used—only the workflow YAMLs themselves | |
| # hold the list | |
| # | |
| name: Quarantined Tests | |
| on: | |
| workflow_dispatch: | |
| schedule: | |
| # Run every 2 hours. Quarantined tests are run frequently to catch flaky tests with a low failure rate. | |
| - cron: '0 */2 * * *' | |
| # Re-enabled with narrow paths filter scoped to key workflow files that | |
| # orchestrate this pipeline. Changes to downstream workflows (e.g. | |
| # build-packages.yml) are validated by the regular CI and don't need to | |
| # re-trigger the full quarantine/outerloop run. | |
| # Previously disabled (#12143) due to broad paths filter causing disk | |
| # space issues on every CI/eng change. | |
| pull_request: | |
| paths: | |
| - '.github/workflows/tests-quarantine.yml' | |
| - '.github/workflows/specialized-test-runner.yml' | |
| - '.github/workflows/run-tests.yml' | |
| - '.github/workflows/build-cli-e2e-image.yml' | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ github.ref }} | |
| cancel-in-progress: true | |
| jobs: | |
| quarantine_tests: | |
| uses: ./.github/workflows/specialized-test-runner.yml | |
| with: | |
| testRunnerName: "QuarantinedTestRunsheetBuilder" | |
| attributeName: "QuarantinedTest" | |
| extraRunSheetBuilderArgs: "-p:RunQuarantinedTests=true" | |
| extraTestArgs: "--filter-trait quarantined=true" | |
| enablePlaywrightInstall: true | |
| ignoreTestFailures: true | |
| # Quarantine passes ignoreTestFailures, so failing quarantined tests do NOT red | |
| # the run (run-tests.yml swallows them and only checks that TRX were produced). | |
| # A failed quarantine run therefore means infrastructure broke — file/append a | |
| # single deduplicated 'automation-broken' issue. No test-failure issues are | |
| # filed here. PR-triggered runs are excluded. | |
| # See docs/ci/specialized-test-failure-issues.md. | |
| report_infra_failure: | |
| name: Report infra failure | |
| needs: [quarantine_tests] | |
| if: ${{ failure() && github.event_name == 'schedule' && github.repository_owner == 'microsoft' }} | |
| runs-on: ubuntu-latest | |
| permissions: | |
| contents: read # checkout to require the local .js modules | |
| actions: read | |
| issues: write | |
| steps: | |
| - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | |
| with: | |
| ref: main | |
| persist-credentials: false | |
| - name: File or update infra issue | |
| uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 | |
| env: | |
| WORKFLOW_FILE: tests-quarantine.yml | |
| DISPLAY_NAME: Quarantined Tests | |
| # A failed quarantine run is always infra (see comment above), so the | |
| # reporter never inspects TRX here. | |
| IGNORE_TEST_FAILURES: 'true' | |
| with: | |
| script: | | |
| const reporter = require('./.github/workflows/report-specialized-test-failures.js'); | |
| await require('./.github/workflows/specialized-test-failure-runner.js')({ github, context, core, reporter }); | |