Feature request: support project dependencies

[catgoose]

Using Neovim 0.10 and playwright 1.44.1

I select 'chromium' using NeotestPlaywrightProject then try to run playwright tests:

My lazy.nvim config:

return {
  "nvim-neotest/neotest",
  dependencies = {
    "nvim-neotest/nvim-nio",
    "nvim-lua/plenary.nvim",
    "antoinemadec/FixCursorHold.nvim",
    "nvim-treesitter/nvim-treesitter",
    "marilari88/neotest-vitest",
    "thenbe/neotest-playwright",
  },
  config = function()
    require("neotest").setup({
      adapters = {
        require("neotest-vitest")({
          filter_dir = function(name)
            return name ~= "node_modules" and name ~= "e2e"
          end,
        }),
        require("neotest-playwright").adapter({
          options = {
            persist_project_selection = true,
            enable_dynamic_test_discovery = true,
            is_test_file = function(file_path)
              return string.match(file_path, "e2e/tests")
            end,
          },
        }),
      },
      consumers = {
        playwright = require("neotest-playwright.consumers").consumers,
      },
    })
  end,
  cmd = {
    "Neotest",
    "NeotestPlaywrightProject",
    "NeotestPlaywrightPreset",
    "NeotestPlaywrightRefresh",
  },
  keys = {
    m("<leader>m", "Neotest summary"),
    m("<leader>n", "Neotest run file"),
  },
}

My playwright.config.ts

...
  reporter: ci
    ? [['list', { printSteps: true }]]
    : [
        ['html', { outputFolder: 'e2e/report' }],
        ['json', { outputFile: 'e2e/test-results.json' }],
      ],
  use: {
    actionTimeout: 0,
    baseURL,
    trace: 'on-first-retry',
    headless: true,
  },
  outputDir: 'e2e/test-results',
  projects: [
    {
      name: 'setup',
      testMatch: msalEnabled ? /.*\.setup\.ts/ : undefined,
    },
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
        storageState: 'e2e/.auth/user.json',
      },
      dependencies,
    },
    {
      name: 'firefox',
      use: {
        ...devices['Desktop Firefox'],
        storageState: 'e2e/.auth/user.json',
      },
      dependencies,
    },
    {
      name: 'webkit',
      use: {
        ...devices['Desktop Safari'],
        storageState: 'e2e/.auth/user.json',
      },
      dependencies,
    },
  ],
...

[catgoose]

Similar to #21 but I can never get them to pass. I am not modifying the test file.

[thenbe]

I wonder if this due to the latest version. Can you reproduce this issue on an older version of playwright (like 1.41.2)?

[catgoose]

I've tried 1.41, 1.40, 1.39 and same issue. It doesn't seem to be updating workspace diagnostics at all. If I add something to a test like:

expect(false).toBeTruthy();

I can view the output of the test, but it does not show diagnostics for that line:

As a comparison here is vitest:

[thenbe]

I'm having difficulty reproducing this.

Btw, you should update your config. it_test_file should be more strict to avoid slurping up unrelated files (e.g. .png from a snapshot test). In your example, it should be something like:

local is_test_file = function(file_path)
	local result = file_path:find('e2e/tests/.*%.test%.[jt]s$') ~= nil
	return result
end

See 985ec26 for more examples.

[catgoose]

Thanks, I added that to my config.

Does neotest-playwright get back the results of a test with the json reporter? Does this have anything to do with the reporter setup in the playwright config? It seems like it's using the list reporter for the output floats in my screenshots.

Here is my full playwright config:

import { defineConfig, devices } from '@playwright/test';
import { config } from 'dotenv';

config();

const baseURL = process.env.PLAYWRIGHT_BASE_URL || 'http://localhost:7373';
const msalEnabled = process.env.MSAL_AUTH_ENABLED === 'true';
const dependencies = msalEnabled ? ['setup'] : [];
const ci = process.env.CI === 'true';

export default defineConfig({
  testDir: './e2e/tests',
  fullyParallel: true,
  timeout: 30 * 1000,
  expect: {
    timeout: 5000,
  },
  forbidOnly: ci,
  retries: ci ? 2 : 0,
  workers: ci ? 1 : undefined,
  reporter: ci
    ? [['list', { printSteps: true }]]
    : [
        ['list', { printSteps: true }],
        ['html', { outputFolder: 'e2e/report', open: 'never' }],
        ['json', { outputFile: 'e2e/test-results.json' }],
      ],
  use: {
    actionTimeout: 0,
    baseURL,
    trace: 'on-first-retry',
    headless: true,
  },
  outputDir: 'e2e/test-results',
  projects: [
    {
      name: 'setup',
      testMatch: msalEnabled ? /.*\.setup\.ts/ : undefined,
    },
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
        storageState: 'e2e/.auth/user.json',
      },
      dependencies,
    },
    {
      name: 'firefox',
      use: {
        ...devices['Desktop Firefox'],
        storageState: 'e2e/.auth/user.json',
      },
      dependencies,
    },
    {
      name: 'webkit',
      use: {
        ...devices['Desktop Safari'],
        storageState: 'e2e/.auth/user.json',
      },
      dependencies,
    },
  ],
  webServer: {
    command: 'npm run dev-ci',
    url: baseURL,
    reuseExistingServer: true,
  },
});

[thenbe]

That might be the cause actually. Apparently I started working on a solution in this PR but never got around to finishing it up.

See this line in particular:

neotest-playwright/README.md

Line 72 in e2ab01c

The only reporter required by `neotest-playwright` is the `json` reporter, which you need to set either in your `playwright.config.ts` or by using `extra_args`. Make sure _not_ to declare the json reporter's `outputFile` property in your config as this will be set by `neotest-playwright`.

If you un-declare the outputFile of the json reporter, is the issue fixed?

-        ['json', { outputFile: 'e2e/test-results.json' }],
+        ['json'],

What about if you remove the json reporter entirely?

[catgoose]

The problem has to do with the dependencies on each project:

    {
      name: 'setup',
      testMatch: msalEnabled ? /.*\.setup\.ts/ : undefined,
    },
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
        storageState: 'e2e/.auth/user.json',
      },
      dependencies: 'setup',
    },

The auth step logs in the user, so I think when that step is completed neotest-playright thinks that the test was successful. If I disable the dependency, I can pass the test:

[catgoose]

Do you know if neotest-playwright can handle dependency projects or should I disable my dependencies temporarily?

[thenbe]

Honestly, I never got around to using dependency projects so I can't say for sure. But given the issue you're facing, some changes might be required in neotest-playwright in order to report the correct results when dependency projects are used.

[catgoose]

I've decided to enable the setup dependency only in CI environment. If I need get auth cookies, I just run the setup project manually to save auth state to user.json.