microsoft/playwright-test

Build a cross-browser end-to-end test suite with Playwright.

Playwright test runner

This project was moved over to the main Playwright repo. See on playwright.dev for the documentation and examples.

playwright-test's Issues

emulate devices

There should be an easy way to specify the device emulation, like an iphone 6 sized viewport with touch support.

great debug mode

We have all of the pieces for a great debugging experience, but we need to tie them together.

• Headful mode
• Run a single test
• Connect to chrome/node via vscode

Browsers not installed

When installing the test runner, the browsers are not installed like they would with playwright.

To repro

• Delete cached browsers directory
• Run npm i -D @playwright/test @playwright/test-runner
• Run npx test-runner which would fail to launch browsers

We could fix this by install playwright in addition to the test runner, but maybe there's a better possibility?

e2e test server

Right now e2e tests require users to set up their own server to serve their website, and we treat it as a black box. It would be nice if we had a default solution that did something useful. We should also have a way to integrate with their existing server.

If we don't treat the server as a black box, we can do a lot of cool features like:

• code coverage that maps to your file system
• nice error message stack traces
• intelligently re-running tests when files are changed
• compiling your code for you just in time
• find your source maps for error locations

The integration would be in the form of a function that takes a localhost url and returns the path to the file on disk which it maps to.

The default reporter eats some of my logs

The default reporter that looks like [1/5]. It overwrites console logs when it updates, which made me very confused.

Hook errors reporting

I'd like to see that afterAll/beforeAll hook has failed, and attribute it to the test case.

Avoid importing fixtures to use it and expect

Currently, every spec file requires importing fixtures to then import it, expect.

// Current
import {fixtures} from '@playwright/test';
const {describe, it, beforeEach, expect} = fixtures;

I suggest we export these objects directly, since most new users don't need to know about fixtures.

// Proposed
import {describe, it, beforeEach, expect} from '@playwright/test';

use playwright-runner with other jest projects

Right now the jest-playwright preset creates two jest projects. Jest does not read nested projects, so that means the preset must be used in a top level config. It also means that you can't extend the preset with other jest options.

We can probably fix this upstream. This issue jestjs/jest#5463 in Jest refers to it.

If that doesn't work, there are some actions we can do in this repo.

1. Merge e2e and unit runners into one runner, so we can export one project, and then somehow figure out which test goes to which runner at runtime.
2. Provide an api in playwirght-runner/index.js that you can include from your jest config. It would take some config, and then return you a jest config merged with the correct playwright options.
3. Provide our own config setup like playwright.config.js for setting things.

Skipping tests

Are there already some thoughts on how the skipping should work?

More a blacklist or a whitelist of browsers?

Jest currently does something like that in their docs, but without adding custom data (for us browsers).

test.skip('it is not snowing', () => {
  expect(inchesOfSnow()).toBe(0);
});

describe.skip('my other beverage', () => {
  // ... will be skipped
});

Maybe something like that

test.skip(["firefox"], 'it is not snowing', () => {
  expect(inchesOfSnow()).toBe(0);
});

describe.skip("firefox", 'my other beverage', () => {
  // ... will be skipped
});

(either an Array or browsers type directly)

Also an interesting mode would be to configure, if the multiple browsers should run in parallel or in order. If its in order, then as proposed in #9 a setup hook can be used to e.g. reset the database.

support jest.mock

This should be doable after #45 is done. Implement jest.mock, using ___webpack_require__ instead of require. babel-plugin-jest-hoist will need to be run on the test files.

setup/teardown

We should have hooks for setting up some test servers or other long running processes when the test run starts and shutting them down when the test run ends. I believe jest already has apis for this, but its a question of integrating with them.

Optionally build test files with Webpack

Modern web dev has a lot of code generated at build time. Instead of trying to parse every combination of commonjs/typescript/scss, people should be able to pass us their webpack config which we will use to compile their unit tests.

Overriding the entry points to contain the test files to be run should provide some native web code we can run.

It would be nice to reuse/cache some shared modules. I'm worried about the performance penalty of compiling a new copy of react into every test file. Maybe webpack already will do this for us?

As a follow up, we can do the same for rollup/browserify/whatever people use.

epilogue prints zero total duration when interrupted

This is what I get after Ctrl+C:

Running 1426 tests using 6 workers
··∘∘····∘∘∘∘∘∘∘····f·················∘∘∘∘∘···············································································∘∘······································∘··································^C
  179 passed (0ms)
  17 skipped

basic-ts broken on @playwright/test 0.9.6

I updated the@playwright/test dev dependency to the latest version 0.9.6 and npx test-runner --browser-name=chromium errors with npm ERR! cb.apply is not a function

https://github.com/microsoft/playwright-test/blob/master/examples/basic-ts/package.json

 "devDependencies": {
    "@playwright/test": "0.9.1"
  },

 npx test-runner --browser-name=chromium
npm ERR! cb.apply is not a function

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/mac/.npm/_logs/2020-09-27T19_49_06_428Z-debug.log
Install for [ 'test-runner@latest' ] failed with code 1

0 info it worked if it ends with ok
1 verbose cli [
1 verbose cli   '/usr/local/Cellar/node/14.12.0/bin/node',
1 verbose cli   '/usr/local/lib/node_modules/npx/node_modules/npm/bin/npm-cli.js',
1 verbose cli   'install',
1 verbose cli   'test-runner@latest',
1 verbose cli   '--global',
1 verbose cli   '--prefix',
1 verbose cli   '/Users/mac/.npm/_npx/15808',
1 verbose cli   '--loglevel',
1 verbose cli   'error',
1 verbose cli   '--json'
1 verbose cli ]
2 info using [email protected]
3 info using [email protected]
4 verbose npm-session e3e77b219de67868
5 silly install loadCurrentTree
6 silly install readGlobalPackageData
7 http fetch GET 304 https://registry.npmjs.org/test-runner 171ms (from cache)
8 silly pacote tag manifest for test-runner@latest fetched in 214ms
9 verbose stack TypeError: cb.apply is not a function
9 verbose stack     at /usr/local/lib/node_modules/npx/node_modules/npm/node_modules/graceful-fs/polyfills.js:287:18
9 verbose stack     at FSReqCallback.oncomplete (fs.js:178:5)
10 verbose cwd /Users/mac/code/playwright/basic-ts
11 verbose Darwin 19.6.0
12 verbose argv "/usr/local/Cellar/node/14.12.0/bin/node" "/usr/local/lib/node_modules/npx/node_modules/npm/bin/npm-cli.js" "install" "test-runner@latest" "--global" "--prefix" "/Users/mac/.npm/_npx/15808" "--loglevel" "error" "--json"
13 verbose node v14.12.0
14 verbose npm  v5.1.0
15 error cb.apply is not a function
16 verbose exit [ 1, true ]

The error starts in version "@playwright/test": "0.9.4". I can update to 0.9.3 without an error.

Links to Examples on README.md point to wrong path

should point to Using JavaScript
but instead it point to
Using JavaScript

screenshot-on-failure option not exists

Take screenshots on failure

npx test-runner --screenshot-on-failure

This example not work, option not exists.

error: unknown option '--screenshot-on-failure'
error Command failed with exit code 1.

node:
"playwright-cli": "next",
"@playwright/test": "^0.9.10",
"@playwright/test-runner": "^0.9.22"

node -v
v12.18.4

it.each and describe.each

describe.each if you keep duplicating the same test suites with different data. describe.each allows you to write the test suite once and pass data in.
https://jestjs.io/docs/en/api#describeeachtablename-fn-timeout

Current fixture union requires exporting both fixture and fixture types. Should only require one.

typescript definitions

Right now there is an @types/jest packages that adds definitions for it/describe/expect into the global scope. This doesn't work for us, because our it has a state as its first argument, instead of a doneCallback.

The unit mode and e2e mode require different types, because the state in e2e mode contains a page, whereas the state in unit mode does not.

Ideally these types wouldn't pollute your apps global scope. So you can only use it from within .spec.js files.

Run in parallel

We should run tests in parallel.

include worker configuration into the failure report

[bug] expect: quotes are escaped inside expected / actual mismatch

When I say
expect('"A"').toBe('"B"')

I see
Expected: ""B""
Received: ""A""

describe not exists or not a function

const { it, describe, beforeAll, beforeEach, afterAll, afterEach } = require('@playwright/test');

describe('feature foo', () => {
  it('is working correctly', async ({ page }) => {
    // Test function
    // ...
  })
});

I get this error

TypeError: describe is not a function

"playwright-cli": "next",
"@playwright/test": "^0.9.10",
"@playwright/test-runner": "^0.9.22"

Provided example does not work

In the example in the README as well in the example in the playwright-runner package directory it's not directly clear for me, what the src/App.spec.jsx does. -> Babel with JSX Plugin e.g. would be needed.

Does the Playwright runner provide the ability to use e.g. ReactDOM.render to interact directly with the DOM in the actual browser or is a development server needed?

cc @JoelEinbinder

focus tests

We need fit and fdescribe.

Type of browserName parameter is any

Expected it to be string or 'chromium' | 'firefox' | 'webkit'

Multiple browsers

Thinking about multiple browsers support.
I suppose it should be configurable, like passing array of browsers:

  browsers: ['chromium', 'webkit']

And test will be passed only if it passed for all browsers.
I've got some thoughts about implementing it. Something like this:

async runTests(testSuites, watcher, onStart, onResult, onFailure, options) {
    // Launching browsers
    const browsers = await Promise.all(_globalConfig.browsers.map(browser => playwright[browser].launch()))
    installGlobals();
    /** @type {WeakMap<Test, import('jest-runner').Test>} */
    const testToSuite = new WeakMap();
    /** @type {Map<any, Set<Test>>} */
    const suiteToTests = new Map();
    const startedSuites = new Set();
    const resultsForSuite = new Map();
    const rootSuite = describe(async () => {
      for (const testSuite of testSuites) {
        const transformer = new ScriptTransformer(testSuite.context.config);
        resultsForSuite.set(testSuite, []);
        suiteToTests.set(testSuite, new Set());
        const suite = describe(async () => {
          transformer.requireAndTranspileModule(testSuite.path);
        });
        for (const test of await suite.tests()) {
          if (testToSuite.has(test))
            continue;
          testToSuite.set(test, testSuite);
          /** @type {Set<Test>} */ (suiteToTests.get(testSuite)).add(test);
        }
      }
    });
    for (const test of await rootSuite.tests()) {
      const suite = /** @type {import('jest-runner').Test} */(testToSuite.get(test));
      if (!startedSuites.has(suite)) {
        startedSuites.add(suite);
        onStart(suite);
      }
      const suiteResults = resultsForSuite.get(suite);
      // Get results for all browsers
      const results = await Promise.all(browsers.map(browser => this._runTest(browser, test)))
      results.map(result => {
        suiteResults.push(result);
        const suiteTests = /** @type {Set<Test>} */ (suiteToTests.get(suite));
        if (suiteTests.size === suiteResults.length)
          onResult(suite, makeSuiteResult(suiteResults, this._globalConfig.rootDir, suite.path));
      })
    }
    purgeRequireCache(testSuites.map(suite => suite.path));
    // Close all browsers
    await Promise.all(browsers.map(browser => browser.close()));
  }

Also it should be able to configure devices with browsers

Cypress <> Playwright Comparison

So these are just quick thoughts on the benefits we get from Cypress, and where Playwright might be able to bridge the gap. Some of these might be things better supported on Playwright itself and not the runner (like proxy config).

These are things that Cypress has that I'm not sure Playwright does yet (correct me if I'm wrong):

1. A headed in-browser test runner UI - developers can watch and debug tests in real time, this is also hooked up to a "watch" mode that picks up code changes. This may be the largest benefit we receive from Cypress in terms of developer experience. The UI test runner allows you to interact with the DOM and pause the test on any steps. (video eg). This UI also surfaces the test failure messages in recorded videos.

2. Config options to run tests against any base URL -- we point Cypress at any host by changing an env variable CYPRESS_BASE_URL.

3. On the same note as the previous ^, ability to propagate any environment variables into browser tests. Cypress uses a convention where any env var prefixed with CYPRESS_ can be discovered and used by tests. https://docs.cypress.io/guides/guides/environment-variables.html#Setting

4. Ability to listen mock and stub routes and API responses https://docs.cypress.io/api/commands/route.html#Syntax

5. Option to output recorded video of the test runs -- seems like you heard this one already #5

6. Proxy configuration for in browser tests https://docs.cypress.io/guides/references/proxy-configuration.html#Set-a-proxy-on-Linux-or-macOS

There's probably more that I'm forgetting, but those are the big ones I can think of.

Auto-waiting assertions

It would be nice to have auto-wait assertions like those in expect-playwright baked in.

https://github.com/playwright-community/expect-playwright

await expect(page).toHaveSelector("#foobar")
await expect(page).toHaveText("#my-element", "MyValue")

[BUG] Pirates is missing

throw err;
^

Error: Cannot find module 'pirates'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/pfeldman/code/playwright-cli/node_modules/@playwright/test-runner/out/transform.js:42:30)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)

setup should behave as a config file

The config pattern like eslintrc.js and jest.config.json makes sense for setup.ts.

Timeout errors

For reference, see microsoft/playwright#2031

This piece of code will not work when running inside Jest:

const playwright = require('playwright');
try {
  await page.waitForSelector('.foo');
} catch (e) {
  if (e instanceof playwright.errors.TimeoutError) {
    // Do something if this is a timeout.
  }
  } else {
    console.log(err);
  }
}

It is possible that the TimeoutError is a different instance because Jest test cases is run inside a vm context, whereas PlaywrightRunnerE2E runs in Node.js.

This does work:

if (err.name === 'TimeoutError')

Note: Have not tested this with playwright-runner, but just assuming this is a problem here as well since it was with jest-playwright.

Add custom Jest matchers

It would be cool to add custom Jest matchers by default to this lib. An example would be expect(page/element).toContainText('some text'), taking into account somehow the 3 different kind of text matchers text=example one (case-insensitive, substring matching), text="example 2" (exact matching), text=/^\\s*Some text$/i (regex matching).

Either in 3 separate functions, or just one matcher with all variants.

Also an expect(page/element).toContainElement (or expect(page/element).toContainSelector which underneath calls waitForSelector.

getting started template

I'd like to have a single command to run like npx setup-playwright that installs all dependencies, adds a jest.config.js, tsconfig.json, github actions, and a sample test.

Watch mode

It would be nice to have a watch mode that automatically re-runs tests when the file is saved similar to cypress. #6

• Allow the user to keep the browser open and close it when the file is re-run
• After hook to allow cleanup (ideally the same ones in #8 #9)

page superpowers

Right now the unit tests can't do anything that a web page can't do. We should add some special playwright apis into the page for testing. Any part of the playwright api that doesn't deal with JSHandles or navigations makes sense to access from unit tests.

• keyboard
• mouse
• route
• accessibility

JavaScript code coverage

Just an issue to sum up whats needed for code coverage.

• A solution which supports all the browsers would probably be hard
• Playwright API already exposes for Chromium v8 raw coverage data
• If the test runner in the end has access to the served files, it could resolve the actual source code via the source maps from my understanding (via the browser like Chromium DevTools is doing, would also be neat. Also less configuration in the end)
• https://github.com/istanbuljs/v8-to-istanbul (something like that)
• Main focus JavaScript test coverage? or also CSS?

beforeAll/afterAll

We should have beforeAll/afterAll hooks. I think they should run before/after the test file, not globally. We should verify that this makes sense with the way most jest tests work.

junit test results

Is it possible to output the test results to a junit format xml file?

Thanks!

export `expect` on the top-level

We can re-export it, but it is handy on the top-level as well.

Record a video for tests

playwright-runner missing feature overview

• allow a --headful CLI parameter to start them headful. Currently the custom CLI args can only be a string.
• allow to use the HEADLESS environment variable as an alias for it. Currently it ends up in max callstack exceed error if you try to redefine them and use the values from the CLI argument. Related to #99?
• allow --browser CLI parameter to define the used browser type. Should work.
• allow the browser be set as an env var. might be related to 2)
• built-in the matrix into the playwright-runner, for device + browser support. Not possible right now afaik.

Action items/tbd:

• I would not define the result of CLI argument as a fixture. Pytest is also not doing that and it seems to be confusing. The alternative would be to expose a fixture called "objects" which contains the options or attach it directly to our "Test" data type.

Screenshots

It would be nice to take a screenshot from a test and compare it to a golden screenshot.

Rerun failures to only rerun failed test cases

Rerunning the whole test file could be slow, I want to jump straight to the failing test case.

fixture.extendWorker/Test was actually useful, can we get it back?

Parameterized tests w/ test.each

It would be nice to support parameterized tests, similar to jest's test.each

each keyword for data-driven test

for data-driven test ,the table parameters is great feature ,so if you can support each keyword ,it will be awesome

Incremental progress reporting

e2e tests are slower, so I'd like to see test cases (or at least their failures) being reported as they run.

[feature] Launch and Context options

relates to #42

tests should still run if one has a syntax error