cypress-io/cypress-test-example-repos

[ARCHIVED] Tests new version of Cypress against multiple projects

cypress-test-example-repos

Testing Cypress Test Runner against different example repos

Build status	Description	YML
	Linux & Mac	circle.yml
	Windows 32-bit	appveyor-win32-x86.yml
	Windows 64-bit	appveyor-win32-x64.yml

Contributing Guideline

Debugging

Run with DEBUG=cypress-test-example-repos environment variable set

License

This project is licensed under the terms of the MIT license.

cypress-test-example-repos's Issues

When testing switch to the branch depending on the version

Imagine we are testing version 1.2.0 and it has breaking changes. Then some of the projects like cypress-example-recipes would have a branch named 1.2.0 already.

Thus when we get a commit message, and extract binary / npm path, we should also extract the version we are testing like 1.2.0, then for each repo see if we can switch to the branch 1.2.0. Then run the tests. If the branch does not exist, no big deal - just run the tests in the default branch.

use newer Docker image

with Chrome and FF browsers

Setup dashboard

Setup a project to record results from all test runs

Setup Windows build on AppVeyor

Can we clone each repo and test on Windows?

Stubbing Google Analytics recipe is failing

Failing all the time ... but when running from this repo

The screenshot on failure image itself looks weird. Why do click + hash + stub look blue?

• https://circleci.com/gh/cypress-io/cypress-test-example-repos/4630
• https://dashboard.cypress.io/#/projects/44775r/runs/d6455ef1-db53-45a1-8d23-723c3feffd91/failures

Investigate issue with Appveyor example-kitchensink

example-kitchensink is failing on appveyor and skipped for now:

cypress-test-example-repos/appveyor-win32-x86.yml

Lines 44 to 46 in e5ce467

	# NOTE: skip for now, failing in appveyor for unknown reason
	# see https://github.com/cypress-io/cypress-test-example-repos/pull/97
	# - npm run test-if -- --repo cypress-example-kitchensink --command test:ci:record:windows

investigate why this started happening and fix it

ESlint settings can interfere with settings in each repo

For example when testing cypress-svelte-unit-test the prettier-standard tool seems to read eslint settings from the root (this repo) when doing its thing, leading to a crash. This crash never happens when testing that repo by itself.

$ npm run pretty

> [email protected] pretty /Users/gleb/git/cypress-test-example-repos/cypress-svelte-unit-test
> prettier-standard 'src/*.js'

prettier-eslint [ERROR]: There was trouble creating the ESLint CLIEngine. Is "eslintPath: /Users/gleb/git/cypress-test-example-repos/cypress-svelte-unit-test/node_modules/eslint" a correct path to the ESLint module?
prettier-standard [ERROR]: There was an error formatting "src/index.js": 
    Error: CLI:
    	Configuration for rule "eqeqeq" is invalid:
    	Value "allow-null,[object Object]" no schemas match.

        at Object.validateRuleOptions (/Users/gleb/git/cypress-test-example-repos/cypress-svelte-unit-test/node_modules/eslint/lib/config/config-validator.js:109:15)
        at CLIEngine.Object.keys.forEach.name (/Users/gleb/git/cypress-test-example-repos/cypress-svelte-unit-test/node_modules/eslint/lib/cli-engine.js:476:19)
        at Array.forEach (<anonymous>)

CI https://circleci.com/gh/cypress-io/cypress-test-example-repos/6678

Should clone repo into temp folder, not into a subfolder

Build unit testing projects

Probably without even moving them into cypress-io organization.

• https://github.com/bahmutov/cypress-vue-unit-test
• https://github.com/bahmutov/cypress-react-unit-test
• https://github.com/bahmutov/cypress-svelte-unit-test
• https://github.com/bahmutov/cypress-cycle-unit-test
• https://github.com/bahmutov/cypress-hyperapp-unit-test

Upgrade to Cypress 3.2.0

enable Windows tests for win32 commits

do not use "prepare" step to skip the windows commits

move CI secure variables into AppVeyor yaml file

To avoid accidentally clearing them again. They should be encrypted

AppVeyor windows CI supports encrypted variables https://ci.appveyor.com/tools/encrypt

NPM install sometimes fails on AppVeyor

Some of the AppVeyor builds seem to break on npm install step

For example, the build https://ci.appveyor.com/project/cypress-io/cypress-test-example-repos/builds/26007256

Author: Renovate Bot <[email protected]>
Date:   Mon Jul 15 05:20:32 2019 +0000
    chore(deps): update to 🌟
rm -rf .git
npm WARN tarball tarball data for [email protected] (sha512-7KxJovlYhTX5RaRbUdkAXN1KUZ8PwWlTzQdHV6xNqvuFOs7+WBo10TQUqT19Q/Jz2hk5v9TQDIhyLhhJY4p5AA==) seems to be corrupted. Trying one more time.
npm ERR! path C:\Users\appveyor\AppData\Local\Temp\1\cypress-example-kitchensink4o2Ay1\node_modules\.staging\typescript-656bcbc4\lib\typescriptServices.js
npm ERR! code EPERM
npm ERR! errno -4048
npm ERR! syscall unlink
npm ERR! Error: EPERM: operation not permitted, unlink 'C:\Users\appveyor\AppData\Local\Temp\1\cypress-example-kitchensink4o2Ay1\node_modules\.staging\typescript-656bcbc4\lib\typescriptServices.js'
npm ERR!  { Error: EPERM: operation not permitted, unlink 'C:\Users\appveyor\AppData\Local\Temp\1\cypress-example-kitchensink4o2Ay1\node_modules\.staging\typescript-656bcbc4\lib\typescriptServices.js'
npm ERR!   cause: 
npm ERR!

Maybe we can retry the npm install command

Bad commit object sha

Seeing for projects that use commit-message-install the following

#!/bin/bash -eo pipefail
$(npm bin)/commit-message-install
last commit: f68174c5383af5df6453dbad1bf54aac339f6dad
fatal: bad object f68174c5383af5df6453dbad1bf54aac339f6dad

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency shelljs to v0.8.5 [SECURITY]
• Update dependency debug to v4.3.3
• Update dependency @bahmutov/print-env to v2
• Update dependency eslint to v8
• Update dependency eslint-plugin-cypress-dev to v3
• Update dependency execa to v6
• Update dependency wait-on to v6
• Click on this checkbox to rebase all open PRs at once

---

• Check this box to trigger a request for Renovate to run again on this repository

Travis-CI + Cypress Issues connecting to site being served using http-server

I have a quick question regarding running Cypress Tests within a CI/CD Pipeline on Travis-CI. I'm currently not able to run my tests as a result of the following error.

'Error: connect ECONNREFUSED'

I'm trying to serve the simple web page I'm trying to test via http-server (https://www.npmjs.com/package/http-server). I've included my travis yaml file below. I'd greatly appreciate any assistance you can provide.

Inside of my test, I'm trying to navigate to the web page using
cy.visit('http://0.0.0.0:1111') and cy.visit('http://localhost:1111') with no luck :(

.travis.yml

language: node_js
node_js:
  - 6
cache:
  directories:
    - ~/.npm
    - node_modules
install:
  - yarn
script:
  - http-server -p 1111 . > http.log 2>&1 &
  - $(npm bin)/cypress run

Setup Mac test build on Buildkite

To check other test projects, same as CircleCI and AppVeyor do

Use commit from message as group id

somehow pass flag to record but with group id taken from the commit message json

Run more example recipes as a test

Bugs in release Cypress v3.5.0 could have been stopped if we ran all cypress-example-recipes against the develop branch (in particular single sign on recipe). Find a way to parallelize the test jobs, probably by running recipes in N jobs with "--chunk" parameter

Test against cypress-example-realworld

Add job to test https://github.com/cypress-io/cypress-example-realworld

Separate 32 and 64 bit AppVeyor projects

Look at the AppVeyor setup in https://github.com/cypress-io/cypress-test-tiny

Find a way to pass group id

Currently we call a command on recipes like this

npm run test:ci

Which in turn calls run-p command

"test:ci": "run-p --race start cypress:run"

Problem - we cannot pass additional CLI arguments. For example npm run test:ci -- --group-id 1 breaks because it calls run-p --race start cypress:run --group-id 1 - which causes run-p to report "unknown option" :(

Need to switch to Node 8

• Cypress itself is Node 8
• Some test projects like cypress-hyperapp-unit-test require build step and parcel-bundler needs 8+

Pass group id and record runs for cypress-example-todomvc repo

npm run test-if -- --repo cypress-example-todomvc --command test:ci

So it gets recorded at https://dashboard.cypress.io/#/projects/44775r/runs for @brian-mann

Switch to using commit-message-install

To install Cypress version from commit message for all projects

See https://github.com/cypress-io/cypress-test-tiny for example

Test against cypress-example-todomvc-redux

Add job to run tests against https://github.com/cypress-io/cypress-example-todomvc-redux

Need to collect artifacts

Because we are testing so many things, if something fails we have no idea why. Like this example https://circleci.com/gh/cypress-io/cypress-test-example-repos/3575#tests/containers/0

It has failed but we do not see the image or video.

  (Screenshots)

  - /home/person/app/cypress-example-recipes/examples/blogs__codepen-demo/cypress/screenshots/HyperApp Counter Codepen -- starts with zero -- before each hook loadAppIFrameAndSetAsOurTestDocument.png (1280x720)


  (Video)

  - Started processing:   Compressing to 32 CRF
  - Finished processing:  /home/person/app/cypress-example-recipes/examples/blogs__codepen-demo/cypress/videos/1yh2u.mp4 (1 second)

Need to collect all generated artifacts after running tests, store on CI

Add testing workshop to list of repos to test against

We really want to keep https://github.com/cypress-io/testing-workshop-cypress up to date without breaking it

Test cypress-dashboard against latest version

Add cypress-dashboard to the list of tested repos.

Move Mac build from Buildkite to CircleCI

lets test cypress-example-kitchensink and cypress-example-todomvc projects only. We are still testing all projects on Linux

• cypress-example-recipes became quite large to always keep testing against
• cypress-example-phonecat and cypress-example-piechopper can be retired

Hmm, times out

At least in some builds like https://circleci.com/gh/cypress-io/cypress-test-example-repos/63184

Starts the job, checks out code, switches to the branch, prints the start of the commit message ... and then just hangs

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Allow testing repos outside cypress-io organization

To test against "guest" repos like "bahmutov/cypress-svelte-unit-test"

$ node ./test-repo.js --repo bahmutov/cypress-svelte-unit-test
set -e
testing url https://github.com/cypress-io/bahmutov/cypress-svelte-unit-test.git
using command test:ci
Cloning into 'cypress-svelte-unit-test'...
remote: Not Found
fatal: repository 'https://github.com/cypress-io/bahmutov/cypress-svelte-unit-test.git/' not found
null
~/git/cypress-test-example-repos on master*
$ echo $?
1

Update Cypress image with the browser

Problem using Chrome browser for some recipes, see https://circleci.com/workflow-run/63e4995c-9d7f-424b-b7e6-a808e148c135