-
-
Notifications
You must be signed in to change notification settings - Fork 8.6k
Add guidance for testing conventions in each language #16734
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
titusfortner
wants to merge
7
commits into
trunk
Choose a base branch
from
testing_guides
base: trunk
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+719
−190
Draft
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
f54b076
Add guidance for testing conventions in each language
titusfortner 844b9ce
update rust
titusfortner 00a62a3
Python updates and additions
cgoldberg 3bc48df
add docs to run tests directly with pytest
navin772 58a693b
Merge branch 'trunk' into testing_guides
navin772 09277b3
run formatter
navin772 392f4cd
Enhance TESTING.md with environment variable details
aguspe File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,134 @@ | ||||||||||||||||
| # .NET Testing Guide | ||||||||||||||||
|
|
||||||||||||||||
| This guide helps contributors write tests in the Selenium .NET codebase. | ||||||||||||||||
|
|
||||||||||||||||
| ## Test Framework | ||||||||||||||||
|
|
||||||||||||||||
| * Tests use NUnit. | ||||||||||||||||
| * All tests inherit from `DriverTestFixture`. | ||||||||||||||||
| * Test HTML pages accessed via properties like `simpleTestPage`, `javascriptPage`. | ||||||||||||||||
| * `WaitFor<T>()` provides waiting with 5-second default timeout. | ||||||||||||||||
|
|
||||||||||||||||
| ```csharp | ||||||||||||||||
| [TestFixture] | ||||||||||||||||
| public class MyFeatureTest : DriverTestFixture | ||||||||||||||||
| { | ||||||||||||||||
| [Test] | ||||||||||||||||
| public void ShouldFindElement() | ||||||||||||||||
| { | ||||||||||||||||
| driver.Url = simpleTestPage; | ||||||||||||||||
| IWebElement element = driver.FindElement(By.Id("foo")); | ||||||||||||||||
| Assert.That(element.Text, Is.EqualTo("expected")); | ||||||||||||||||
| } | ||||||||||||||||
|
|
||||||||||||||||
| [Test] | ||||||||||||||||
| [IgnoreBrowser(Browser.Safari, "Safari doesn't support this")] | ||||||||||||||||
| public void ShouldDoSomething() | ||||||||||||||||
| { | ||||||||||||||||
| // Skipped on Safari | ||||||||||||||||
| } | ||||||||||||||||
| } | ||||||||||||||||
| ``` | ||||||||||||||||
|
|
||||||||||||||||
| ## Running Tests | ||||||||||||||||
|
|
||||||||||||||||
| Bazel creates test targets for each browser. Always use `--pin_browsers`. | ||||||||||||||||
RenderMichael marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||||||||
|
|
||||||||||||||||
| ```shell | ||||||||||||||||
| bazel test //dotnet/test/common:AllTests --pin_browsers=true # Default browser (Firefox) | ||||||||||||||||
| bazel test //dotnet/test/common:AllTests-chrome --pin_browsers=true | ||||||||||||||||
| bazel test //dotnet/test/common:AllTests-firefox --pin_browsers=true | ||||||||||||||||
| bazel test //dotnet/test/common:AllTests-edge --pin_browsers=true | ||||||||||||||||
|
|
||||||||||||||||
| # Additional Arguments | ||||||||||||||||
| bazel test //dotnet/... --flaky_test_attempts=3 --pin_browsers=true | ||||||||||||||||
| bazel test //dotnet/... --test_output=all --pin_browsers=true | ||||||||||||||||
| ``` | ||||||||||||||||
|
|
||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A few lines about running tests in the IDE.
Suggested change
|
||||||||||||||||
| ## Attributes | ||||||||||||||||
|
|
||||||||||||||||
| ### Skipping Tests | ||||||||||||||||
|
|
||||||||||||||||
| | Attribute | When to Use | | ||||||||||||||||
| |-----------|-------------| | ||||||||||||||||
| | `[IgnoreBrowser(Browser.X, "reason")]` | Skip test for specific browser | | ||||||||||||||||
| | `[IgnorePlatform("windows", "reason")]` | Skip test on specific OS | | ||||||||||||||||
| | `[IgnoreTarget("net8", "reason")]` | Skip test on specific .NET version | | ||||||||||||||||
| | `[Ignore("reason")]` | Skip test entirely (NUnit built-in) | | ||||||||||||||||
|
|
||||||||||||||||
| ```csharp | ||||||||||||||||
| [Test] | ||||||||||||||||
| [IgnoreBrowser(Browser.Safari, "Safari doesn't support multiple instances")] | ||||||||||||||||
| [IgnoreBrowser(Browser.IE, "IE is flaky")] | ||||||||||||||||
| public void TestWithMultipleDrivers() | ||||||||||||||||
| { | ||||||||||||||||
| } | ||||||||||||||||
|
|
||||||||||||||||
| [Test] | ||||||||||||||||
| [IgnorePlatform("windows", "Thread time not supported")] | ||||||||||||||||
| public void TestLinuxOnly() | ||||||||||||||||
| { | ||||||||||||||||
| } | ||||||||||||||||
| ``` | ||||||||||||||||
|
|
||||||||||||||||
| Browser values: `Browser.Chrome`, `Browser.Firefox`, `Browser.Edge`, `Browser.Safari`, `Browser.IE`, `Browser.Remote`, `Browser.All` | ||||||||||||||||
|
|
||||||||||||||||
| ### Driver Lifecycle | ||||||||||||||||
|
|
||||||||||||||||
| | Attribute | When to Use | | ||||||||||||||||
| |-----------|-------------| | ||||||||||||||||
| | `[NeedsFreshDriver(IsCreatedBeforeTest = true)]` | Fresh driver before test | | ||||||||||||||||
| | `[NeedsFreshDriver(IsCreatedAfterTest = true)]` | Fresh driver after test | | ||||||||||||||||
|
|
||||||||||||||||
| ```csharp | ||||||||||||||||
| [Test] | ||||||||||||||||
| [NeedsFreshDriver(IsCreatedBeforeTest = true, IsCreatedAfterTest = true)] | ||||||||||||||||
| [IgnoreBrowser(Browser.Safari, "Safari doesn't support multiple instances")] | ||||||||||||||||
| public void TestRequiringFreshDriver() | ||||||||||||||||
| { | ||||||||||||||||
| IWebDriver driver2 = EnvironmentManager.Instance.CreateDriverInstance(); | ||||||||||||||||
| try | ||||||||||||||||
| { | ||||||||||||||||
| // Test with multiple drivers | ||||||||||||||||
| } | ||||||||||||||||
| finally | ||||||||||||||||
| { | ||||||||||||||||
| driver2.Quit(); | ||||||||||||||||
| } | ||||||||||||||||
| } | ||||||||||||||||
| ``` | ||||||||||||||||
|
|
||||||||||||||||
| ## Helpers | ||||||||||||||||
|
|
||||||||||||||||
| From `DriverTestFixture`: | ||||||||||||||||
|
|
||||||||||||||||
| | Member | Description | | ||||||||||||||||
| |--------|-------------| | ||||||||||||||||
| | `driver` | Current WebDriver instance | | ||||||||||||||||
| | `simpleTestPage`, `javascriptPage`, etc. | Test page URLs | | ||||||||||||||||
| | `WaitFor<T>(condition, timeout)` | Wait for condition (default 5s) | | ||||||||||||||||
| | `CreateFreshDriver()` | Create new driver instance | | ||||||||||||||||
|
|
||||||||||||||||
| From `EnvironmentManager.Instance`: | ||||||||||||||||
|
|
||||||||||||||||
| | Member | Description | | ||||||||||||||||
| |--------|-------------| | ||||||||||||||||
| | `CreateDriverInstance()` | Create additional driver | | ||||||||||||||||
| | `CreateDriverInstance(options)` | Create driver with custom options | | ||||||||||||||||
| | `Browser` | Current browser enum value | | ||||||||||||||||
|
|
||||||||||||||||
| ## Test Organization | ||||||||||||||||
|
|
||||||||||||||||
| ``` | ||||||||||||||||
| dotnet/test/ | ||||||||||||||||
| ├── common/ # Cross-browser tests | ||||||||||||||||
| │ ├── DriverTestFixture.cs # Base class | ||||||||||||||||
| │ ├── CustomTestAttributes/ # Custom NUnit attributes | ||||||||||||||||
| │ └── *Test.cs # Test files | ||||||||||||||||
| └── support/ # Support library tests | ||||||||||||||||
| ``` | ||||||||||||||||
|
|
||||||||||||||||
| ## Build Files | ||||||||||||||||
|
|
||||||||||||||||
| * Adding tests shouldn't require Bazel changes—tests are picked up via glob. | ||||||||||||||||
| * Make sure `*Test.cs` files are in a directory with `dotnet_nunit_test_suite` in BUILD.bazel. | ||||||||||||||||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hyperlinking to each binding readme would also be beneficial.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is a good idea. I was going to do that, but our READMEs aren't consistent right now with what they do and don't contain, and I was thinking it might make sense to update them to mostly point to our website docs, and right now the coding agents we're targeting can't access the internet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added an issue to track this - #16740