-
Notifications
You must be signed in to change notification settings - Fork 2.6k
docs(nx-dev): document playwright output merging w/ atomizer #32830
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
4c7b29c
docs(nx-dev): add playwright output merging guide
barbados-clemens 5d1ce82
docs(nx-dev): add playwright atomized merged output to nx.dev
barbados-clemens d53286c
docs(nx-dev): apply PR feedback
barbados-clemens 6d8e39d
docs(nx-dev): add playwright atomized merged output to nx.dev
barbados-clemens File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
9 changes: 9 additions & 0 deletions
9
astro-docs/src/content/docs/technologies/test-tools/playwright/Guides/index.mdoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| --- | ||
| title: Guides | ||
| sidebar: | ||
| hidden: true | ||
| description: Playwright configuration and usage | ||
| pagefind: false | ||
| --- | ||
|
|
||
| {% index_page_cards path="technologies/test-tools/playwright/guides" /%} |
231 changes: 231 additions & 0 deletions
231
...cs/src/content/docs/technologies/test-tools/playwright/Guides/merge-atomized-outputs.mdoc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,231 @@ | ||
| --- | ||
| title: Merge Atomized Playwright Outputs | ||
| description: Learn how to enable output merging for atomized Playwright tests | ||
| sidebar: | ||
| label: Merge Atomized Outputs | ||
| filter: 'type:Guides' | ||
| --- | ||
|
|
||
| When running Playwright tests in CI environments, Nx can [split tests across multiple parallel tasks](/docs/features/ci-features/split-e2e-tasks) for faster execution. | ||
| However, this creates separate test reports for each task. | ||
| With the Nx Plugin, the Playwright report merging feature allows you to combine these individual reports into a single, unified report. | ||
|
|
||
| **Benefits:** | ||
|
|
||
| - View all test results in one consolidated report | ||
| - Maintain the same reporting experience as non-atomized tests | ||
| - Keep the performance benefits of parallel test execution | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Nx workspace with `@nx/playwright` plugin | ||
| - Playwright tests configured in your project | ||
| - CI environment with [task atomization enabled](/docs/features/ci-features/split-e2e-tasks) | ||
| - Nx v21.6.1 | ||
|
|
||
| {% aside type="note" title="Nx Version"%} | ||
| Nx v21.6.1 introduced automatic configuration of report merging which is the focus of this guide. | ||
| Similar steps can be followed for manual configuration of Playwright for any versions prior to Nx v21.6.1 | ||
| {%/aside%} | ||
|
|
||
| ## How report merging works | ||
|
|
||
| 1. **Blob Reports**: Playwright generates [lightweight "blob" reports](https://playwright.dev/docs/test-reporters#blob-reporter) during test execution | ||
| 2. **Individual Tasks**: Each atomized test task creates its own blob report | ||
| 3. **Report Merging**: A separate merge task combines all blob reports into unified output using your configured reporters (HTML, JSON, JUnit, etc.) | ||
|
|
||
| ## Configuration | ||
|
|
||
| ### Step 1: Enable blob reports | ||
|
|
||
| The `nxE2EPreset` from `@nx/playwright/preset` automatically enable the blob reporter when running in CI: | ||
|
|
||
| ```ts | ||
| // playwright.config.ts | ||
| import { defineConfig } from '@playwright/test'; | ||
| import { nxE2EPreset } from '@nx/playwright/preset'; | ||
|
|
||
| export default defineConfig({ | ||
| // automatically uses the blob reporter for CI | ||
| ...nxE2EPreset(__filename, { testDir: './src' }), | ||
| // Your additional configuration | ||
| }); | ||
| ``` | ||
|
|
||
| > The preset has a `generateBlobReports` option that can be used to disable the reporter or maybe enable it not just in CI. | ||
|
|
||
| **Alternative manual configuration:** | ||
|
|
||
| ```ts | ||
| // playwright.config.ts | ||
| export default defineConfig({ | ||
| reporter: [ | ||
| ['html', { outputFolder: 'playwright-report' }], | ||
| ...(process.env.CI ? [['blob', { outputDir: 'blob-report' }]] : []), | ||
| ], | ||
| // Your other configuration | ||
| }); | ||
| ``` | ||
|
|
||
| ### Step 2: Configure the Playwright plugin | ||
|
|
||
| Add a `ciTargetName` to the Playwright plugin to your `nx.json`. This will enable task splitting and report merging features for CI environments: | ||
|
|
||
| ```json | ||
| // nx.json | ||
| { | ||
| "plugins": [ | ||
| { | ||
| "plugin": "@nx/playwright/plugin", | ||
| "options": { | ||
| "targetName": "e2e", | ||
| "ciTargetName": "e2e-ci" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| With the above configurations, Nx automatically creates these targets for your Playwright projects: | ||
|
|
||
| - `e2e`: Regular target for local development (runs all tests together) | ||
| - `e2e-ci`: CI target that runs tests in parallel across multiple tasks | ||
| - `e2e-ci--merge-reports`: Target that merges blob reports into unified reports | ||
|
|
||
| ## Running the feature | ||
|
|
||
| **Local Development:** | ||
|
|
||
| ```shell {% frame="none" %} | ||
| # Run tests normally (no blob reports generated) | ||
| nx run my-app:e2e | ||
| ``` | ||
|
|
||
| **CI Environment:** | ||
|
|
||
| ```shell {% frame="none" %} | ||
| # Run atomized tests (generates blob reports) | ||
| nx run my-app:e2e-ci | ||
|
|
||
| # Merge the reports | ||
| nx run my-app:e2e-ci--merge-reports | ||
| ``` | ||
|
|
||
| ## Complete example | ||
|
|
||
| Here's a complete working example for reference: | ||
|
|
||
| ```ts | ||
| // playwright.config.ts | ||
| import { defineConfig } from '@playwright/test'; | ||
| import { nxE2EPreset } from '@nx/playwright/preset'; | ||
|
|
||
| export default defineConfig({ | ||
| ...nxE2EPreset(__filename, { testDir: './e2e' }), | ||
| use: { | ||
| baseURL: 'http://localhost:4200', | ||
| }, | ||
| webServer: { | ||
| command: 'npm run start', | ||
| port: 4200, | ||
| reuseExistingServer: !process.env.CI, | ||
| }, | ||
| }); | ||
| ``` | ||
|
|
||
| ```json | ||
| // nx.json | ||
| { | ||
| "plugins": [ | ||
| { | ||
| "plugin": "@nx/playwright/plugin", | ||
| "options": { | ||
| "targetName": "e2e", | ||
| "ciTargetName": "e2e-ci" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| ## CI Integration | ||
|
|
||
| > The following example CI configuration is for GitHub Actions, but the same steps can be followed in all CI providers. | ||
| > The most important part is to always call the merge report target, `e2e-ci--merge-reports`, even if the E2E tests fail to always have a final merged report. | ||
|
|
||
| ```yaml | ||
| // .github/workflows/ci.yaml | ||
| name: CI | ||
| on: | ||
| pull_request: | ||
|
|
||
| jobs: | ||
| main: | ||
| steps: | ||
| ... | ||
| - name: Run E2E Tests | ||
| run: nx run my-app:e2e-ci | ||
|
|
||
| - name: Merge Test Reports | ||
| run: nx run my-app:e2e-ci--merge-reports | ||
| # Run even if tests fail to get partial results | ||
| if: always() | ||
| ``` | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### "The blob reporter is not configured" | ||
|
|
||
| **Problem:** The merge task warns that no blob reporter is configured. | ||
|
|
||
| **Solution:** Ensure the blob reporter is added to your Playwright configuration: | ||
|
|
||
| - If overriding the `reporter` in the Playwright configuration, make sure to add the blob reporter: | ||
|
|
||
| ```ts | ||
| // playwright.config.ts | ||
| export default defineConfig({ | ||
| reporter: [ | ||
| // existing reporters | ||
| ...(process.env.CI ? [['blob', { outputDir: 'blob-report' }]] : []), | ||
| ], | ||
| // Your other configuration | ||
| }); | ||
| ``` | ||
|
|
||
| ### Missing test results in report | ||
|
|
||
| **Problem:** The merge shows fewer results than expected test suites. | ||
|
|
||
| **Solution:** | ||
|
|
||
| - Check if some test tasks failed or were cancelled | ||
| - Verify all blob report files are present in the blob report directory | ||
| - Review CI logs for any task failures | ||
|
|
||
| ### Reports not merging | ||
|
|
||
| **Problem:** Individual blob reports exist but aren't being merged. | ||
|
|
||
| **Solution:** | ||
|
|
||
| - Ensure the merge target is running after all test tasks complete | ||
| - The default merge target name is `e2e-ci--merge-reports` | ||
| - You can run `nx show project <e2e-project-name>` to view targets available on your project to verify the target exists | ||
| - Check that the blob report directory path is correct | ||
| - Verify Playwright CLI is available in the CI environment | ||
|
|
||
| ### [WARNING] "No additional reporters are configured" | ||
|
|
||
| **Problem:** Only the blob reporter is configured, so it will be the only report produced | ||
|
|
||
| **Solution:** Add at least one additional reporter (HTML, JSON, JUnit, etc.) alongside the blob reporter. | ||
|
|
||
| List of available Playwright reports can be found in the [Playwright documentation](https://playwright.dev/docs/test-reporters). | ||
|
|
||
| ## Best Practices | ||
|
|
||
| 1. **Environment-based Configuration**: Only enable blob reports in CI to avoid overhead during local development | ||
| 2. **Error Handling**: Use `if: always()` (or similar) in CI workflows to merge partial results even when some tests fail | ||
| 3. **Output Artifacts**: Archive both blob reports and merged reports as CI artifacts for debugging | ||
| 4. **Validation**: The merge task automatically validates the expected number of test results | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.