Skip to content

Flaky Tests Policies [DOCS-10820] #30887

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions content/en/tests/flaky_management/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,23 @@ Use the status drop-down to change how a flaky test is handled in your CI pipeli

<div class="alert alert-info"><strong>Note</strong>: Status actions have minimum version requirements for each programming language's instrumentation library. See <a href="#compatibility">Compatibility</a> for details.</div>

## Configure policies to automate the flaky test lifecycle

Configure automated Flaky Tests Policies to govern how flaky tests are handled in each repository. For example, a test that flakes in the default branch can automatically be quarantined, and later disabled if it remains unfixed after 30 days.

1. Click the **Policies** button at the upper right of the Flaky Management page. You can also navigate to [**Flaky Tests Policies**][11] in Software Delivery settings.
2. Search for and select the repository you want to configure. This opens the **Edit Policies** flyout.
{{< img src="tests/flaky-policies.png" alt="Flaky Tests Policies page with the Edit Policies flyout open to configure a policy" style="width:100%;" >}}

3. Use the toggles to enable specific automated actions, and use automation rules to further customize how tests get quarantined, disabled, or retried:

| Action | Description |
| ---- | ---- |
| **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <li>Time: Quarantine a test if its status is Active for a specified number of days. <li>Branch: Quarantine an Active test if it flakes in one or more specified branches.|
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency with the rest of the document where statuses use inline code

Suggested change
| **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <li>Time: Quarantine a test if its status is Active for a specified number of days. <li>Branch: Quarantine an Active test if it flakes in one or more specified branches.|
| **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <li>Time: Quarantine a test if its status is `Active` for a specified number of days. <li>Branch: `Quarantine` an `Active` test if it flakes in one or more specified branches.|

| **Disable** | Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness. Customize automation rules based on: <li>Status and time: Disable a test if it has a specified status for a specified number of days. <li>Branch: Disable an Active or Quarantined test if it flakes in one or more specified branches. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| **Disable** | Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness. Customize automation rules based on: <li>Status and time: Disable a test if it has a specified status for a specified number of days. <li>Branch: Disable an Active or Quarantined test if it flakes in one or more specified branches. |
| **Disable** | Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness. Customize automation rules based on: <li>Status and time: Disable a test if it has a specified status for a specified number of days. <li>Branch: Disable an `Active` or `Quarantined` test if it flakes in one or more specified branches. |

| **Attempt&nbsp;to&nbsp;Fix** | When you attempt to fix a flaky test, automatically retry the test a specified number of times on the commit containing the fix. |
| **Fixed** | If a flaky test no longer flakes for 30 days, it is automatically moved to Fixed status. This automation is default behavior and can't be customized. |

## Track evolution of flaky tests

Track the evolution of the number of flaky tests with the `test_optimization.test_management.flaky_tests` out-of-the-box metric. The metric is enriched with the tags below to help you investigate the counts in more detail.
Expand Down Expand Up @@ -126,3 +143,4 @@ To use Flaky Test Management features, you must use Datadog's native instrumenta
[8]: /tests/setup/javascript/
[9]: /tests/setup/python/
[10]: /tests/setup/ruby/
[11]: https://app.datadoghq.com/ci/settings/test-optimization/flaky-test-management
Binary file added static/images/tests/flaky-policies.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading