Possibility to define browser

netlify-plugin-cypress

Runs Cypress end-to-end tests on Netlify Build

Install and use

You can install this plugin in the Netlify UI from this direct in-app installation link or from the Plugins directory.

For file based installation, add netlify-plugin-cypress NPM package as a dev dependency to your repository.

npm install --save-dev netlify-plugin-cypress
# or
yarn add -D netlify-plugin-cypress

And then add the plugin's name to the list of build plugins in netlify.toml file as shown in the examples below.

note: this plugin assumes you have already installed Cypress as a dev NPM dependency.

Chromium install

This plugin installs via Puppeteer Chromium browser, which is also cached inside ./node_modules folder.

How does it work

Build steps

When Netlify Build system runs it performs 2 steps essentially:

builds the site
deploys the site

Every plugin that wants to perform some actions can do so before the build, after the build (but before the deploy), and after the deploy. The Netlify uses the following names for these events

"preBuild"
1. builds the site
"postBuild"
2. deploys the site
"onSuccess"
"onFailure"

Thus every plugin can register itself to be executed before a site is built using "preBuild" event, or after a successful deploy using "onSuccess" event name, etc.

This plugin

This plugin netlify-plugin-cypress by default runs during the "onSuccess" event, testing the deployed site. The Netlify Build system gives the URL to the plugin and it runs Cypress against that URL using the Cypress NPM module API.

Optionally, you can also run tests during "preBuild" and "postBuild" steps. This is useful if you want to ensure the site is working even before deploying it to Netlify servers. Finally, this plugin does not use "onFailure" event which happens only if Netlify fails to deploy the site.

Failing the deploy

Running Cypress tests by default uses the "onSuccess" step of the build pipeline. By this point Netlify has already deployed the site. Even if the tests fail now, the Netlify shows the successful deployment - the site is live! To really prevent the broken deploys, we suggest using Cypress GitHub / GitLab / Bitbucket integration to fail the status checks on a pull request.

We also suggest running tests during the "preBuild" and/or "postBuild" steps. If the tests fail during these steps, the Netlify fails the entire build and does not deploy the broken site.

Finally, you can set up Slack notifications on failed tests against the deployed site. At least you will quickly find out if the deployed site fails the E2E tests and would be able to roll back the deploy.

Examples

basic

Here is the most basic Netlify config file netlify.toml with just the Cypress plugin

[[plugins]]
  # runs Cypress tests against the deployed URL
  package = "netlify-plugin-cypress"

The example file above should be enough to run Cypress tests in any existing Netlify project.

# explicit commands for building the site
# and the folder to publish
[build]
command = "npm run build"
publish = "build"

[build.environment]
# cache Cypress binary in local "node_modules" folder
# so Netlify caches it
CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
# set TERM variable for terminal output
TERM = "xterm"

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"

See netlify-plugin-cypress-example repo.

Typescript users may need to add a install before the build command. For a yarn user with a typescript app, the build section of the Netlify configuration might look like this:

[build]
command = "yarn install && yarn build"
publish = "build"

# ...remaining configuration...

tutorial

Read the full tutorial at Test Sites Deployed To Netlify Using netlify-plugin-cypress.

Note: if any tests against the deployed URL fail, the Netlify build still considers it a success. Thus if you want to have a test check against the deploy, install Cypress GitHub App. The app will provide its own failing status check in this case.

options

You can control the browser, the specs to run, record tests on Cypress Dashboard, etc, see manifest.yml file.

recording

To record test results and artifacts on Cypress Dashboard, set record: true plugin input and set CYPRESS_RECORD_KEY as an environment variable via Netlify Deploy settings.

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  record = true

See cypress-example-kitchensink and recorded results at Cypress Dashboard

Security note 🔐: you should keep your CYPRESS_RECORD_KEY secret. You can control how Netlify builds external pull requests, see the doc - you never want to expose sensitive environment variables to outside builds.

status checks

If you are recording test results to Cypress Dashboard, you should also install Cypress GitHub Integration App to see status checks from individual groups or from individual specs per commit. See netlify-plugin-prebuild-example PR #8 pull request for an example.

group

You can change the group name for the recorded run using group parameter

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  record = true
  group = "built site"

tag

You can give recorded run tags using a comma-separated string. If the tag is not specified, Netlify context will be used (production, deploy-preview or branch-deploy)

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  record = true
  group = "built site"
  tag = "nightly,production"

spec

Run only a single spec or specs matching a wildcard

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
# runs Cypress tests against the deployed URL
package = "netlify-plugin-cypress"
  [plugins.inputs]
  spec = "cypress/integration/smoke*.js"

See cypress-example-kitchensink for instance.

browser

By default all tests run using the Chromium browser. If you want to use Electron:

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
package = "netlify-plugin-cypress"
  [plugins.inputs]
  # allowed values: electron, chromium
  browser = "electron"

configFile

If you would like to use a different Cypress config file instead of cypress.json, specify it using the configFile option

[build]
command = "npm run build"
publish = "build"
  [build.environment]
  # cache Cypress binary in local "node_modules" folder
  # so Netlify caches it
  CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary"
  # set TERM variable for terminal output
  TERM = "xterm"

[[plugins]]
package = "netlify-plugin-cypress"
  [plugins.inputs]
  configFile = "cypress.netlify.config.js"

testing SPA routes

SPAs need catch-all redirect setup to make non-root paths accessible by tests. You can enable this with spa parameter.

[[plugins]]
# local Cypress plugin will test our site after it is built
package = "netlify-plugin-cypress"
  [plugins.inputs]
  # can also use "spa = true" to use "index.html" by default
  spa = "index.html"

See lws-spa for more options and tests/routing example.

testing the site before build

By default this plugin tests static site after deploy. But maybe you want to run end-to-end tests against the local development server. You can start the local server, wait for it to respond and then run Cypress tests by passing parameters to this plugin. Here is a sample config file

[[plugins]]
  package = "netlify-plugin-cypress"
  # let's run tests against development server
  # before building it (and testing the built site)
  [plugins.inputs.preBuild]
    enable = true
    start = 'npm start'
    wait-on = 'http://localhost:3000'
    wait-on-timeout = '30' # seconds

Parameters you can place into preBuild inputs: start, wait-on, wait-on-timeout, spec, record, group, and tag.

See netlify-plugin-prebuild-example repo

testing the site after build

By default this plugin tests static site after deploy. But maybe you want to run end-to-end tests locally after building the static site. Cypress includes a local static server for this case but you can specify your own command if needed by using the start argument. Here is a sample config file

[[plugins]]
  package = "netlify-plugin-cypress"
  # let's run tests against the built site
  [plugins.inputs.postBuild]
    enable = true

Parameters you can place into postBuild inputs: spec, record, group, tag, start and spa.

The SPA parameter

If your site requires all unknown URLs to redirect back to the index page, use the spa parameter

[[plugins]]
  package = "netlify-plugin-cypress"
  # let's run tests against the built site
  [plugins.inputs.postBuild]
    enable = true
    # must allow our test server to redirect unknown routes to "/"
    # so that client-side routing can correctly route them
    # can be set to true or "index.html" (or similar fallback filename in the built folder)
    spa = true
    start = 'npm start'

See the routing example.

using Netlify CLI

Even better when testing the prebuilt site is to run the Netlify CLI to make sure the local API redirects and Netlify functions work in addition to the web site. Add netlify-cli as a dev dependency and start it during testing.

$ npm i -D netlify-cli

[[plugins]]
  package = "netlify-plugin-cypress"
  # start Netlify server
  [plugins.inputs.preBuild]
    start = 'npx netlify dev'
    wait-on = 'http://localhost:8888'

For more, see tests/test-netlify-dev example and read Testing Netlify Function section.

skipping tests

If you are testing the site before building it and want to skip testing the deployed URL

[[plugins]]
  package = "netlify-plugin-cypress"
  # do not test the deployed URL
  [plugins.inputs]
    enable = false
  # test the local site
  [plugins.inputs.preBuild]
    enable = true

parallelization

Running tests in parallel is not supported because Netlify plugin system runs on a single machine. Thus you can record the tests on Cypress Dashboard, but not run tests in parallel. If Netlify expands its build offering by allowing multiple build machines, we could take advantage of it and run tests in parallel.

HTML files

When serving the built folder, we automatically serve .html files. For example, if your folder has the following structure:

public/
  index.html
  pages/
    about.html

The public folder is served automatically and the following test successfully visits both the root and the about.html pages:

cy.visit('/')
cy.visit('/pages/about') // visits the about.html

Example repos

Name	Description
netlify-plugin-cypress-example	Runs Cypress tests on Netlify and records their results to Cypress Dashboard
netlify-plugin-prebuild-example	Runs tests twice, first using the development version of the site, then after Netlify builds the production bundles, runs the tests again
cypress-example-kitchensink	Runs only a subset of all tests before publishing the folder to Netlify
bahmutov/eleventyone	Example used in Test Sites Deployed To Netlify Using netlify-plugin-cypress tutorial
gatsby-starter-portfolio-cara	A Gatsby site example

Major upgrades

v1 to v2

The default browser has been switched to Chromium. If you want to use the built-in Electron use an explicit option browser
We have changed the default testing phase. In v1 the tests executed after building the site by default. In v2 the tests run against the deployed URL by default, and you need to enable the testing during preBuild or postBuild steps.

Debugging

Set environment variable DEBUG=netlify-plugin-cypress to see the debug logs. To see even more information, set DEBUG=netlify-plugin-cypress,netlify-plugin-cypress:verbose

Warning: be careful with verbose logging, since it can print all environment variables passed to the plugin, including tokens, API keys, and other secrets.

Common problems

Too many progress messages while installing Cypress

If you see A LOT of progress messages during "npm install" step, set an environment variable during build CI = 1 to remove them.

Cypress binary is installed on every build

By default Cypress binary is installed in the home folder, see caching. Netlify build does NOT cache this folder, but it DOES cache the local "node_modules" folder. Tell Cypress to install its binary in the "node_modules" folder by setting build environment variable CYPRESS_CACHE_FOLDER = "./node_modules/CypressBinary".

Several versions of Cypress are installed according to the build logs

From the Netlify UI under Deploys, pick "Trigger Deploy" and select "Clear cache and deploy site". This should cleanly install new "node_modules" and remove old Cypress versions.

Term message warnings in the Cypress output

If you see messages like tput: No value for $TERM and no -T specified during Cypress run, add an environment variable TERM = xterm.

Electron browser crashes while running tests

Switch to using Chromium browser that seems to be a bit more reliable. Use browser = "chromium" setting.

You want to skip Puppeteer download

If you do not plan on using Chromium to run the tests, if you want to use the built-in Electron browser, you can save time by skipping the Puppeteer download. Set the environment variable PUPPETEER_SKIP_DOWNLOAD = 1 on your CI.

License

This project is licensed under the terms of the MIT license.

Contributing

Read the contributing guide

	async function postBuild({ fullPublishFolder, record, spec, group, tag, buildUtils }) {
	const port = 8080
	const server = serveFolder(fullPublishFolder, port)
	debug('local server listening on port %d', port)

	const baseUrl = `http://localhost:${port}`

	const results = await runCypressTests(baseUrl, record, spec, group, tag)

	await new Promise((resolve, reject) => {
	server.close(err => {
	if (err) {
	return reject(err)
	}
	debug('closed local server on port %d', port)
	resolve()
	})
	})

	processCypressResults(results, buildUtils)
	}

cypress-io / netlify-plugin-cypress Goto Github PK

netlify-plugin-cypress's Introduction