-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Flaky Tests Policies [DOCS-10820] #30887
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
joepeeples
wants to merge
3
commits into
master
Choose a base branch
from
joepeeples/10820-flaky-test-policies
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -38,6 +38,23 @@ Use the status drop-down to change how a flaky test is handled in your CI pipeli | |||||
|
||||||
<div class="alert alert-info"><strong>Note</strong>: Status actions have minimum version requirements for each programming language's instrumentation library. See <a href="#compatibility">Compatibility</a> for details.</div> | ||||||
|
||||||
## Configure policies to automate the flaky test lifecycle | ||||||
|
||||||
Configure automated Flaky Tests Policies to govern how flaky tests are handled in each repository. For example, a test that flakes in the default branch can automatically be quarantined, and later disabled if it remains unfixed after 30 days. | ||||||
|
||||||
1. Click the **Policies** button at the upper right of the Flaky Management page. You can also navigate to [**Flaky Tests Policies**][11] in Software Delivery settings. | ||||||
2. Search for and select the repository you want to configure. This opens the **Edit Policies** flyout. | ||||||
{{< img src="tests/flaky-policies.png" alt="Flaky Tests Policies page with the Edit Policies flyout open to configure a policy" style="width:100%;" >}} | ||||||
|
||||||
3. Use the toggles to enable specific automated actions, and use automation rules to further customize how tests get quarantined, disabled, or retried: | ||||||
|
||||||
| Action | Description | | ||||||
| ---- | ---- | | ||||||
| **Quarantine** | Toggle to allow flaky tests to be quarantined for this repository. Customize automation rules based on: <li>Time: Quarantine a test if its status is Active for a specified number of days. <li>Branch: Quarantine an Active test if it flakes in one or more specified branches.| | ||||||
| **Disable** | Toggle to allow flaky tests to be disabled for this repository. You may want to do this after quarantining or to protect specific branches from flakiness. Customize automation rules based on: <li>Status and time: Disable a test if it has a specified status for a specified number of days. <li>Branch: Disable an Active or Quarantined test if it flakes in one or more specified branches. | | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
| **Attempt to Fix** | When you attempt to fix a flaky test, automatically retry the test a specified number of times on the commit containing the fix. | | ||||||
| **Fixed** | If a flaky test no longer flakes for 30 days, it is automatically moved to Fixed status. This automation is default behavior and can't be customized. | | ||||||
|
||||||
## Track evolution of flaky tests | ||||||
|
||||||
Track the evolution of the number of flaky tests with the `test_optimization.test_management.flaky_tests` out-of-the-box metric. The metric is enriched with the tags below to help you investigate the counts in more detail. | ||||||
|
@@ -126,3 +143,4 @@ To use Flaky Test Management features, you must use Datadog's native instrumenta | |||||
[8]: /tests/setup/javascript/ | ||||||
[9]: /tests/setup/python/ | ||||||
[10]: /tests/setup/ruby/ | ||||||
[11]: https://app.datadoghq.com/ci/settings/test-optimization/flaky-test-management |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For consistency with the rest of the document where statuses use inline code