Automate End 2 End Tests

[jasonbahl]

Currently we're running E2E tests manually, because we test against a live WP Engine site.

It would be nice to have a Github workflow that uploads the zip of the plugin to a live WPE site, then runs the tests.

Questions:

• Since the tests need to happen in a live environment, how to we ensure we don't have multiple tests running at the same time, trying to upload different versions of the same plugin(s)?

• If we have a PR to WPGraphQL or WPGraphQL Smart Cache that we need to test against, how do we ensure we're uploading that version of WPGraphQL / WPGraphQL Smart Cache to test against?

  • for example, I'm working on a PR to WPGraphQL core to refactor options to support option invalidation. We'll want to run the e2e tests with that version of WPGraphQL

• What should trigger the tests? Opening a PR to this repo? Does it need a specific label or tag or branch naming convention (i.e. release/x.x.x)

[jasonbahl]

One idea could be to use some environment variables to specify what fork of WPGraphQL / WPGraphQL Smart Cache and what branch to use.

For example, if I open a PR to WPGraphQL and we want to test against it, we could use an env var like:

WP_GRAPHQL_REPO="https://github.com/jasonbahl/wp-graphql"< my fork
WP_GRAPHQL_BRANCH="feature/add-some-filter-for-caching" < my pr branch

defaults of these variables could be something like:

WP_GRAPHQL_REPO="https://github.com/wp-graphql/wp-graphql"
WP_GRAPHQL_BRANCH="master"

but we could override them to test against different forks as needed 🤔

[markkelnar]

Things to figure out

• credentials for the WP test user stored in github
• can plugin zip be uploaded to the WP via the codeception end-2-end test suite? or some other tool?
• Can the smart-cache plugin be uploaded from that github repo? Can specific version or PR be uploaded?
• What about the race condition of the tests running via more than one github action runner and colliding on the version of the plugins running and the state of the WP?

[jasonbahl]

We might also need to consider a matrix as well so we know what versions of what plugins cause conflicts with each other? 🤔

Like is xx plugin incompatible with xx version of WPGraphQL?