Merge branch 'develop' into release/15.0.0

Author: jennifer-shehane

.circleci/workflows.yml

@@ -1,7 +1,7 @@
 version: 2.1
 
 chrome-stable-version: &chrome-stable-version "135.0.7049.114"
-chrome-beta-version: &chrome-beta-version "136.0.7103.33"
+chrome-beta-version: &chrome-beta-version "136.0.7103.48"
 firefox-stable-version: &firefox-stable-version "137.0"
 
 orbs:

cli/CHANGELOG.md

@@ -21,8 +21,18 @@ _Released 07/01/2025 (PENDING)_
 
 _Released 5/6/2025 (PENDING)_
 
+**Performance:**
+
+- Ensure the previous pausing event handlers are removed before new ones are added. Addressed in [#31596](https://github.com/cypress-io/cypress/pull/31596).
+
+**Bugfixes:**
+
+- Fixed an issue where the configuration setting `trashAssetsBeforeRuns=false` was ignored for assets in the `videosFolder`. These assets were incorrectly deleted before running tests with `cypress run`. Addresses [#8280](https://github.com/cypress-io/cypress/issues/8280).
+- Fixed a potential hang condition when navigating to `about:blank`. Addressed in [#31634](https://github.com/cypress-io/cypress/pull/31634).
+
 **Misc:**
 
+- The Assertions menu when you right click in `experimentalStudio` tests now displays in dark mode. Addresses [#10621](https://github.com/cypress-io/cypress-services/issues/10621). Addressed in [#31598](https://github.com/cypress-io/cypress/pull/31598).
 - The URL in the Cypress App no longer displays a white background when the URL is loading. Fixes [#31556](https://github.com/cypress-io/cypress/issues/31556).
 
 ## 14.3.2

cli/types/cypress.d.ts

@@ -3324,7 +3324,6 @@ declare namespace Cypress {
     spec: Cypress['spec'] | null
     specs: Array<Cypress['spec']>
     isDefaultProtocolEnabled: boolean
-    isStudioProtocolEnabled: boolean
     hideCommandLog: boolean
     hideRunnerUi: boolean
   }

guides/README.md

@@ -19,7 +19,9 @@ For general contributor information, check out [`CONTRIBUTING.md`](../CONTRIBUTI
 * [Error handling](./error-handling.md)
 * [GraphQL Subscriptions - Overview and Test Guide](./graphql-subscriptions.md)
 * [Patching packages](./patch-package.md)
+* [Protocol development](./protocol-development.md)
 * [Release process](./release-process.md)
+* [Studio development](./studio-development.md)
 * [Testing other projects](./testing-other-projects.md)
 * [Testing strategy and style guide (draft)](./testing-strategy-and-styleguide.md)
 * [Writing cross-platform JavaScript](./writing-cross-platform-javascript.md)

guides/studio-development.md

@@ -1,14 +1,33 @@
 # Studio Development
 
-In production, the code used to facilitate Studio functionality will be retrieved from the Cloud. However, in order to develop locally, developers will:
+In production, the code used to facilitate Studio functionality will be retrieved from the Cloud. While Studio is still in its early stages it is hidden behind an environment variable: `CYPRESS_ENABLE_CLOUD_STUDIO` but can also be run against local cloud Studio code via the environment variable: `CYPRESS_LOCAL_STUDIO_PATH`.
+
+To run against locally developed Studio:
 
 - Clone the `cypress-services` repo (this requires that you be a member of the Cypress organization)
   - Run `yarn`
-  - Run `yarn watch` in `app/studio`
-- Set `CYPRESS_LOCAL_STUDIO_PATH` to the path to the `cypress-services/app/studio/dist/development` directory
+  - Run `yarn watch` in `app/packages/studio`
+- Set:
+  - `CYPRESS_INTERNAL_ENV=<environment>` (e.g. `staging` or `production` if you want to hit those deployments of `cypress-services` or `development` if you want to hit a locally running version of `cypress-services`)
+  - `CYPRESS_LOCAL_STUDIO_PATH` to the path to the `cypress-services/app/packages/studio/dist/development` directory
+ 
+To run against a deployed version of studio:
+
+- Set:
+  - `CYPRESS_INTERNAL_ENV=<environment>` (e.g. `staging` or `production` if you want to hit those deployments of `cypress-services` or `development` if you want to hit a locally running version of `cypress-services`)
+  - `CYPRESS_ENABLE_CLOUD_STUDIO=true`
+
+Regardless of running against local or deployed studio:
+
 - Clone the `cypress` repo
   - Run `yarn`
   - Run `yarn cypress:open`
+  - Log In to the Cloud via the App
+  - Ensure the project has been setup in the `Cypress (staging)` if in staging environment or `Cypress Internal Org` if in production environment and has a `projectId` that represents that. If developing against locally running `cypress-services`, ensure that the project has the feature `studio-ai` enabled for it.
+  - Open a project that has `experimentalStudio: true` set in the `e2e` config of the `cypress.config.js|ts` file.
+  - Click to 'Add Commands to Test' after hovering over a test command.
+
+Note: When using the `CYPRESS_LOCAL_STUDIO_PATH` environment variable or when running the Cypress app via the locally cloned repository, we bypass our error reporting and instead log errors to the browser or node console.
 
 ## Types
 
@@ -23,3 +42,59 @@ or to reference a local `cypress_services` repo:
 ```sh
 CYPRESS_LOCAL_STUDIO_PATH=<path-to-cypress-services/app/studio/dist/development-directory> yarn gulp downloadStudioTypes
 ```
+
+## Testing
+
+### Unit/Component Testing
+
+The code that supports cloud Studio and lives in the `cypress` monorepo is unit and component tested in a similar fashion to the rest of the code in the repo. See the [contributing guide](https://github.com/cypress-io/cypress/blob/ad353fcc0f7fdc51b8e624a2a1ef4e76ef9400a0/CONTRIBUTING.md?plain=1#L366) for more specifics.
+
+The code that supports cloud Studio and lives in the `cypress-services` monorepo has unit and component tests that live alongside the code in that monorepo.
+
+### Cypress in Cypress Testing
+
+Several helpers are provided to facilitate testing cloud Studio using Cypress in Cypress tests. The [helper file](https://github.com/cypress-io/cypress/blob/ad353fcc0f7fdc51b8e624a2a1ef4e76ef9400a0/packages/app/cypress/e2e/studio/helper.ts) provides a method, `launchStudio` that:
+
+1. Loads a project (by default [`experimental-studio`](https://github.com/cypress-io/cypress/tree/develop/system-tests/projects/experimental-studio)).
+2. Navigates to the appropriate spec (by default `specName.cy.js`).
+3. Enters Studio either by creating a new test or entering from an existing test via the `createNewTest` parameter
+4. Waits for the test to finish executing again in Studio mode.
+
+The above steps actually download the studio code from the cloud and use it for the test. Note that `experimental-studio` is set up to be a `canary` project so it will always get the latest and greatest of the cloud Studio code, whether or not it has been fully promoted to production. Note that this means that if you are writing Cypress in Cypress tests that depend on new functionality delivered from the cloud, the Cypress in Cypress tests cannot be merged until the code lands and is built in the cloud. Local development is still possible however by setting `process.env.CYPRESS_LOCAL_STUDIO_PATH` to your local studio path where we enable studio [here](https://github.com/cypress-io/cypress/blob/develop/packages/frontend-shared/cypress/e2e/e2ePluginSetup.ts#L424).
+
+In order to properly engage with Studio AI, we choose to simulate the cloud interactions that enable it via something like:
+
+```js
+cy.mockNodeCloudRequest({
+  url: '/studio/testgen/n69px6/enabled',
+  method: 'get',
+  body: { enabled: true },
+})
+```
+
+To ensure that we get the same results from our Studio AI calls every time, we simulate them via something like:
+
+```js
+const aiOutput = 'cy.get(\'button\').should(\'have.text\', \'Increment\')'
+cy.mockNodeCloudStreamingRequest({
+  url: '/studio/testgen/n69px6/generate',
+  method: 'post',
+  body: { recommendations: [{ content: aiOutput }] },
+})
+```
+
+The above two helpers actually mock out the Node requests so we still test the interface between the browser and node with these tests.
+
+Also, since protocol does not work properly on the inner Cypress of Cypress in Cypress tests, we choose to create a dummy protocol which means we need to provide a simulated CDP full snapshot that will be sent to AI via something like:
+
+```js
+cy.mockStudioFullSnapshot({
+  id: 1,
+  nodeType: 1,
+  nodeName: 'div',
+  localName: 'div',
+  nodeValue: 'div',
+  children: [],
+  shadowRoots: [],
+})
+```

packages/app/cypress/e2e/cypress-in-cypress-e2e.cy.ts

@@ -214,8 +214,7 @@ describe('Cypress In Cypress E2E', { viewportWidth: 1500, defaultCommandTimeout:
     cy.specsPageIsVisible()
     cy.contains('withFailure.spec').click()
     cy.contains('[aria-controls=reporter-inline-specs-list]', 'Specs')
-    // A bit of a hack, but our cy-in-cy test needs to wait for the reporter to fully render before pressing the "f" key to expand the "Search specs" menu.
-    // Otherwise, the "f" keypress happens before the event is registered, which causes the "Search Specs" menu to not expand.
+
     cy.get('[data-cy="runnable-header"]').should('be.visible')
     cy.get('body').type('f')
     cy.contains('Search specs')

packages/app/cypress/e2e/reporter_header.cy.ts

@@ -11,6 +11,7 @@ describe('Reporter Header', () => {
     })
 
     it('selects the correct spec in the Specs List', () => {
+      cy.get('[data-cy="runnable-header"]').should('be.visible')
       cy.get('body').type('f')
 
       cy.get('[data-selected-spec="true"]').should('contain', 'dom-content').should('have.length', '1')
@@ -19,6 +20,7 @@ describe('Reporter Header', () => {
 
     // TODO: Reenable as part of https://github.com/cypress-io/cypress/issues/23902
     it.skip('filters the list of specs when searching for specs', () => {
+      cy.get('[data-cy="runnable-header"]').should('be.visible')
       cy.get('body').type('f')
 
       cy.findByTestId('specs-list-panel').within(() => {

packages/app/cypress/e2e/runner/event-manager.cy.ts

@@ -1,11 +1,11 @@
-import { loadSpec } from './support/spec-loader'
+import { loadSpec, shouldHaveTestResults } from './support/spec-loader'
 
 describe('event-manager', () => {
   it('emits the cypress:created event when spec is rerun', () => {
     // load the spec initially
     loadSpec({
       filePath: 'hooks/basic.cy.js',
-      passCount: 1,
+      passCount: 2,
     })
 
     cy.window().then((win) => {
@@ -26,4 +26,28 @@ describe('event-manager', () => {
       cy.wrap(() => eventReceived).invoke('call').should('be.true')
     })
   })
+
+  it('clears the pause listeners when the spec is rerun', () => {
+    loadSpec({
+      filePath: 'hooks/basic.cy.js',
+      passCount: 2,
+    })
+
+    cy.window().then((win) => {
+      const eventManager = win.getEventManager()
+
+      cy.wrap(() => eventManager.reporterBus.listeners('runner:next').length).invoke('call').should('equal', 1)
+
+      // trigger a rerun
+      cy.get('.restart').click()
+
+      shouldHaveTestResults({
+        passCount: 2,
+        failCount: 0,
+        pendingCount: 0,
+      })
+
+      cy.wrap(() => eventManager.reporterBus.listeners('runner:next').length).invoke('call').should('equal', 1)
+    })
+  })
 })

packages/app/cypress/e2e/runner/pluginEvents.cy.ts

@@ -57,6 +57,7 @@ describe('plugin events', () => {
         })
       })
 
+      cy.get('[data-cy="runnable-header"]').should('be.visible')
       cy.get('body').type('f')
       cy.get('div[title="run_events_spec_2.cy.js"]').click()
       cy.waitForSpecToFinish({

packages/app/cypress/e2e/runner/sessions.ui.cy.ts

@@ -772,6 +772,7 @@ describe('runner/cypress sessions.open_mode.spec', () => {
   })
 
   it('persists global session and does not persists spec session when selecting a different spec', () => {
+    cy.get('[data-cy="runnable-header"]').should('be.visible')
     cy.get('body').type('f')
     cy.get('div[title="blank_session.cy.js"]').click()
 

packages/app/cypress/e2e/specs_list_component.cy.ts

@@ -27,6 +27,7 @@ describe('App: Spec List (Component)', () => {
   it('highlights the currently running spec', () => {
     cy.contains('fails').click()
     cy.contains('[aria-controls=reporter-inline-specs-list]', 'Specs')
+    cy.get('[data-cy="runnable-header"]').should('be.visible')
     cy.get('body').type('f')
     cy.get('[data-selected-spec="true"]').should('contain', 'fails')
     cy.get('[data-selected-spec="false"]').should('contain', 'foo')

packages/app/cypress/e2e/specs_list_e2e.cy.ts

@@ -121,6 +121,7 @@ describe('App: Spec List (E2E)', () => {
 
       cy.contains('[aria-controls=reporter-inline-specs-list]', 'Specs')
       cy.findByText('Your tests are loading...').should('not.be.visible')
+      cy.get('[data-cy="runnable-header"]').should('be.visible')
       cy.get('body').type('f')
 
       cy.get('[data-selected-spec="true"]').contains('dom-content.spec.js')
@@ -136,8 +137,6 @@ describe('App: Spec List (E2E)', () => {
       cy.findByText('Your tests are loading...').should('not.be.visible')
 
       cy.contains('[aria-controls=reporter-inline-specs-list]', 'Specs')
-      // A bit of a hack, but our cy-in-cy test needs to wait for the reporter to fully render before pressing the "f" key to expand the "Search specs" menu.
-      // Otherwise, the "f" keypress happens before the event is registered, which causes the "Search Specs" menu to not expand.
       cy.get('[data-cy="runnable-header"]').should('be.visible')
       // open the inline spec list
       cy.get('body').type('f')

packages/app/cypress/e2e/studio/studio.cy.ts

@@ -56,11 +56,52 @@ describe('studio functionality', () => {
 
       cy.window().then((win) => {
         expect(win.Cypress.config('isDefaultProtocolEnabled')).to.be.false
-        expect(win.Cypress.config('isStudioProtocolEnabled')).to.be.true
         expect(win.Cypress.state('isProtocolEnabled')).to.be.true
       })
     })
 
+    it('loads the studio UI correctly when studio bundle is taking too long to load', () => {
+      loadProjectAndRunSpec({ enableCloudStudio: false })
+
+      cy.window().then(() => {
+        cy.withCtx((ctx) => {
+          // Mock the studioLifecycleManager.getStudio method to return a hanging promise
+          if (ctx.coreData.studioLifecycleManager) {
+            const neverResolvingPromise = new Promise<null>(() => {})
+
+            ctx.coreData.studioLifecycleManager.getStudio = () => neverResolvingPromise
+            ctx.coreData.studioLifecycleManager.isStudioReady = () => false
+          }
+        })
+      })
+
+      cy.contains('visits a basic html page')
+      .closest('.runnable-wrapper')
+      .findByTestId('launch-studio')
+      .click()
+
+      cy.waitForSpecToFinish()
+
+      // Verify the cloud studio panel is not present
+      cy.findByTestId('studio-panel').should('not.exist')
+
+      cy.get('[data-cy="loading-studio-panel"]').should('not.exist')
+
+      cy.get('[data-cy="hook-name-studio commands"]').should('exist')
+
+      cy.getAutIframe().within(() => {
+        cy.get('#increment').realClick()
+      })
+
+      cy.findByTestId('hook-name-studio commands').closest('.hook-studio').within(() => {
+        cy.get('.command').should('have.length', 2)
+        cy.get('.command-name-get').should('contain.text', '#increment')
+        cy.get('.command-name-click').should('contain.text', 'click')
+      })
+
+      cy.get('button').contains('Save Commands').should('not.be.disabled')
+    })
+
     it('does not display Studio button when not using cloud studio', () => {
       loadProjectAndRunSpec({ })
 
@@ -126,6 +167,74 @@ describe('studio functionality', () => {
 
       cy.percySnapshot()
     })
+
+    it('opens a cloud studio session with AI enabled', () => {
+      cy.mockNodeCloudRequest({
+        url: '/studio/testgen/n69px6/enabled',
+        method: 'get',
+        body: { enabled: true },
+      })
+
+      const aiOutput = 'cy.get(\'button\').should(\'have.text\', \'Increment\')'
+
+      cy.mockNodeCloudStreamingRequest({
+        url: '/studio/testgen/n69px6/generate',
+        method: 'post',
+        body: { recommendations: [{ content: aiOutput }] },
+      })
+
+      cy.mockStudioFullSnapshot({
+        id: 1,
+        nodeType: 1,
+        nodeName: 'div',
+        localName: 'div',
+        nodeValue: 'div',
+        children: [],
+        shadowRoots: [],
+      })
+
+      const deferred = pDefer()
+
+      loadProjectAndRunSpec({ enableCloudStudio: true })
+
+      cy.findByTestId('studio-panel').should('not.exist')
+
+      cy.intercept('/cypress/e2e/index.html', () => {
+        // wait for the promise to resolve before responding
+        // this will ensure the studio panel is loaded before the test finishes
+        return deferred.promise
+      }).as('indexHtml')
+
+      cy.contains('visits a basic html page')
+      .closest('.runnable-wrapper')
+      .findByTestId('launch-studio')
+      .click()
+
+      // regular studio is not loaded until after the test finishes
+      cy.get('[data-cy="hook-name-studio commands"]').should('not.exist')
+      // cloud studio is loaded immediately
+      cy.findByTestId('studio-panel').then(() => {
+        // check for the loading panel from the app first
+        cy.get('[data-cy="loading-studio-panel"]').should('be.visible')
+        // we've verified the studio panel is loaded, now resolve the promise so the test can finish
+        deferred.resolve()
+      })
+
+      cy.wait('@indexHtml')
+
+      // Studio re-executes spec before waiting for commands - wait for the spec to finish executing.
+      cy.waitForSpecToFinish()
+
+      // Verify the studio panel is still open
+      cy.findByTestId('studio-panel')
+      cy.get('[data-cy="hook-name-studio commands"]')
+
+      // Verify that AI is enabled
+      cy.get('[data-cy="ai-status-text"]').should('contain.text', 'Enabled')
+
+      // Verify that the AI output is correct
+      cy.get('[data-cy="studio-ai-output-textarea"]').should('contain.text', aiOutput)
+    })
   })
 
   it('updates an existing test with an action', () => {