Component test runner in cypress open-ct mode throws error when collecting coverage

[cee-chen]

Logs and screenshots

Versions

• What is this plugin's version? 3.9.11
• If the plugin worked before in version X, but stopped after upgrading to version Y, please try the released versions between X and Y to see where the breaking change was.
  • N/A, this is fresh code. Just installed code-coverage for the first time a few hours ago
• What is Cypress version? 8.3.0
• What is your operating system? Mac 11.6
• What is the shell? zsh
• What is the Node version? v14.17.6
• What is the NPM version? 6.14.15
• How do you instrument your application? Cypress does not instrument web application code, so you need to do it yourself.
  • babel-plugin-instanbul - I was following the instructions of https://github.com/bahmutov/cypress-and-jest#cypress-code-coverage-setup
• When running tests, if you open the web application in regular browser, and open DevTools, do you see window.__coverage__ object? Can you paste a screenshot?
  • Currently undefined, likely due to Cypress throwing/crashing before it can be defined
  • 
• Is there .nyc_output folder? Is there .nyc_output/out.json file. Is it empty? Can you paste at least part of it so we can see the keys and file paths?
  • .nyc_output.out.json exists and is not empty, which means code coverage is correctly being calcuated. I don't think that's relevant to the problem/error being thrown however so I'm going to opt not to paste the entire file, but I can upon request
• Do you have any custom NYC settings in package.json (nyc object) or in other NYC config files

  "nyc": {
    "report-dir": "reports/cypress-coverage",
    "reporter": [
      "json",
      "html"
    ]
  },
  

• Do you run Cypress tests in a Docker container?
  • We will on CI, but I'm not even at that stage yet. This is failing locally on dev

Describe the bug

Hey y'all. Cypress code coverage is working great right now for component testing when run via cypress run-ct and correct calculates and displays coverage, but fails/throws in interactive mode with an open browser (cypress open-ct).

In terms of the component test runner, we use neither CRA nor Vue so our plugins setup uses the Generic Webpack code to set up the component dev server (below code screenshot is from the Cypress doc linked in the previous paragraph):

Here is our extremely basic bones cypress/plugins/index.js which essentially contains the exact boilerplate code from https://github.com/cypress-io/code-coverage#instrument-unit-tests and the above screenshot:

module.exports = (on, config) => {
  require('@cypress/code-coverage/task')(on, config);
  on('file:preprocessor', require('@cypress/code-coverage/use-babelrc'));

  if (config.testingType === 'component') {
    const { startDevServer } = require('@cypress/webpack-dev-server');

    // Your project's Webpack configuration
    const webpackConfig = require('./webpack.config.js');

    on('dev-server:start', (options) =>
      startDevServer({ options, webpackConfig })
    );
  }

  return config;
};

1. If I comment out require('@cypress/code-coverage/task')(on, config), cypress open-ct once again starts working with no issues (but obviously with no code coverage - it logs a beforeAll warning message of ⚠️ Code coverage tasks were not registered by the plugins file).

2. If I comment out return config, cypress open-ct bizarrely starts working again but also still fails to log or output code coverage. run-ct also still works without the returned config somehow 🤷‍♀️. My suspicion is that something is going wrong with how the code coverage task modifies config and how startDevServer uses that returned config.

Thankfully this works on run-ct so I'll just use that for now and manually re-run coverage reports after every change for now, but it would be great to have open-ct working with code coverage for obvious reasons (hot reloading, easier to inspect DOM for failed tests, etc.)

Link to the repo
Here's my working diff where I set up our existing Cypress component test runner config with code coverage:
elastic/eui@master...constancecchen:8268384

In terms of repo installation, you should be able to clone, yarn, and then yarn test-cypress-dev to open the test runner.

[cee-chen]

Also wanted to mention that I did spend a decent amount of time looking at related issues to see if they could provide a fix or workaround, but to no avail:

• twhoff's workaround in Component tests with create-react-app (react-scripts-v4) don't instrument #461 (comment) is for CRA, not a generic webpack project, and additionally does not appear to be using the interactive component test runner (no testingType check)

• Error: Code coverage tasks not registered by the plugins file #405 seems slightly related in that our config.env.codeCoverageTasksRegistered is also true but failing, but in the end does not seem specific to the component test runner either (and I'm not sure how I would merge/combine configs in this case)

[cee-chen]

Argh, well I discovered today that I'm an idiot - disregard this issue totally 😅

What the actual totally unrelated error was: I was copying a few it() tests over from Jest to Cypress to retain the test names, and those tests had async in them, e.g. it('does something', async () => {}).

The async (even though there was no corresponding await) was what was causing the Cypress detected that you returned a promise from a command while also invoking one or more cy commands in that promise error. Incidentally, it was also all causing my tests to pass as successes in run-ct mode when some should have been failing, which is what tipped me off in the end 🤦‍♀️

Total f-up on my part, Cypress code coverage is now working great in both run-ct and open-ct modes with no issues. Hey, maybe this'll help someone out there with the same issue someday!