microsoft / playwright-test Goto Github PK
View Code? Open in Web Editor NEWBuild a cross-browser end-to-end test suite with Playwright.
Home Page: https://playwright.dev
License: Apache License 2.0
Build a cross-browser end-to-end test suite with Playwright.
Home Page: https://playwright.dev
License: Apache License 2.0
Right now there is an @types/jest
packages that adds definitions for it
/describe
/expect
into the global scope. This doesn't work for us, because our it
has a state as its first argument, instead of a doneCallback.
The unit mode and e2e mode require different types, because the state in e2e mode contains a page
, whereas the state in unit mode does not.
Ideally these types wouldn't pollute your apps global scope. So you can only use it
from within .spec.js
files.
Right now the jest-playwright preset creates two jest projects. Jest does not read nested projects, so that means the preset must be used in a top level config. It also means that you can't extend the preset with other jest options.
We can probably fix this upstream. This issue jestjs/jest#5463 in Jest refers to it.
If that doesn't work, there are some actions we can do in this repo.
playwright.config.js
for setting things.When I say
expect('"A"').toBe('"B"')
I see
Expected: ""B""
Received: ""A""
Right now the unit tests can't do anything that a web page can't do. We should add some special playwright apis into the page for testing. Any part of the playwright api that doesn't deal with JSHandles or navigations makes sense to access from unit tests.
I updated the@playwright/test
dev dependency to the latest version 0.9.6
and npx test-runner --browser-name=chromium
errors with npm ERR! cb.apply is not a function
https://github.com/microsoft/playwright-test/blob/master/examples/basic-ts/package.json
"devDependencies": {
"@playwright/test": "0.9.1"
},
npx test-runner --browser-name=chromium
npm ERR! cb.apply is not a function
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/mac/.npm/_logs/2020-09-27T19_49_06_428Z-debug.log
Install for [ 'test-runner@latest' ] failed with code 1
0 info it worked if it ends with ok
1 verbose cli [
1 verbose cli '/usr/local/Cellar/node/14.12.0/bin/node',
1 verbose cli '/usr/local/lib/node_modules/npx/node_modules/npm/bin/npm-cli.js',
1 verbose cli 'install',
1 verbose cli 'test-runner@latest',
1 verbose cli '--global',
1 verbose cli '--prefix',
1 verbose cli '/Users/mac/.npm/_npx/15808',
1 verbose cli '--loglevel',
1 verbose cli 'error',
1 verbose cli '--json'
1 verbose cli ]
2 info using [email protected]
3 info using [email protected]
4 verbose npm-session e3e77b219de67868
5 silly install loadCurrentTree
6 silly install readGlobalPackageData
7 http fetch GET 304 https://registry.npmjs.org/test-runner 171ms (from cache)
8 silly pacote tag manifest for test-runner@latest fetched in 214ms
9 verbose stack TypeError: cb.apply is not a function
9 verbose stack at /usr/local/lib/node_modules/npx/node_modules/npm/node_modules/graceful-fs/polyfills.js:287:18
9 verbose stack at FSReqCallback.oncomplete (fs.js:178:5)
10 verbose cwd /Users/mac/code/playwright/basic-ts
11 verbose Darwin 19.6.0
12 verbose argv "/usr/local/Cellar/node/14.12.0/bin/node" "/usr/local/lib/node_modules/npx/node_modules/npm/bin/npm-cli.js" "install" "test-runner@latest" "--global" "--prefix" "/Users/mac/.npm/_npx/15808" "--loglevel" "error" "--json"
13 verbose node v14.12.0
14 verbose npm v5.1.0
15 error cb.apply is not a function
16 verbose exit [ 1, true ]
The error starts in version "@playwright/test": "0.9.4"
. I can update to 0.9.3
without an error.
This is what I get after Ctrl+C:
Running 1426 tests using 6 workers
··∘∘····∘∘∘∘∘∘∘····f·················∘∘∘∘∘···············································································∘∘······································∘··································^C
179 passed (0ms)
17 skipped
We should have hooks for setting up some test servers or other long running processes when the test run starts and shutting them down when the test run ends. I believe jest already has apis for this, but its a question of integrating with them.
should point to Using JavaScript
but instead it point to
Using JavaScript
The default reporter that looks like [1/5]
. It overwrites console logs when it updates, which made me very confused.
Just an issue to sum up whats needed for code coverage.
for data-driven test ,the table parameters is great feature ,so if you can support each keyword ,it will be awesome
We have all of the pieces for a great debugging experience, but we need to tie them together.
This should be doable after #45 is done. Implement jest.mock
, using ___webpack_require__
instead of require
. babel-plugin-jest-hoist
will need to be run on the test files.
It would be nice to support parameterized tests, similar to jest's test.each
There should be an easy way to specify the device emulation, like an iphone 6 sized viewport with touch support.
When installing the test runner, the browsers are not installed like they would with playwright.
To repro
npm i -D @playwright/test @playwright/test-runner
npx test-runner
which would fail to launch browsersWe could fix this by install playwright in addition to the test runner, but maybe there's a better possibility?
Modern web dev has a lot of code generated at build time. Instead of trying to parse every combination of commonjs/typescript/scss, people should be able to pass us their webpack config which we will use to compile their unit tests.
Overriding the entry points to contain the test files to be run should provide some native web code we can run.
It would be nice to reuse/cache some shared modules. I'm worried about the performance penalty of compiling a new copy of react into every test file. Maybe webpack already will do this for us?
As a follow up, we can do the same for rollup/browserify/whatever people use.
We should have beforeAll/afterAll hooks. I think they should run before/after the test file, not globally. We should verify that this makes sense with the way most jest tests work.
--headful
CLI parameter to start them headful. Currently the custom CLI args can only be a string.HEADLESS
environment variable as an alias for it. Currently it ends up in max callstack exceed error if you try to redefine them and use the values from the CLI argument. Related to #99?--browser
CLI parameter to define the used browser type. Should work.Action items/tbd:
In the example in the README as well in the example in the playwright-runner package directory it's not directly clear for me, what the src/App.spec.jsx
does. -> Babel with JSX Plugin e.g. would be needed.
Does the Playwright runner provide the ability to use e.g. ReactDOM.render to interact directly with the DOM in the actual browser or is a development server needed?
I'd like to see that afterAll/beforeAll hook has failed, and attribute it to the test case.
Thinking about multiple browsers support.
I suppose it should be configurable, like passing array of browsers:
browsers: ['chromium', 'webkit']
And test will be passed only if it passed for all browsers.
I've got some thoughts about implementing it. Something like this:
async runTests(testSuites, watcher, onStart, onResult, onFailure, options) {
// Launching browsers
const browsers = await Promise.all(_globalConfig.browsers.map(browser => playwright[browser].launch()))
installGlobals();
/** @type {WeakMap<Test, import('jest-runner').Test>} */
const testToSuite = new WeakMap();
/** @type {Map<any, Set<Test>>} */
const suiteToTests = new Map();
const startedSuites = new Set();
const resultsForSuite = new Map();
const rootSuite = describe(async () => {
for (const testSuite of testSuites) {
const transformer = new ScriptTransformer(testSuite.context.config);
resultsForSuite.set(testSuite, []);
suiteToTests.set(testSuite, new Set());
const suite = describe(async () => {
transformer.requireAndTranspileModule(testSuite.path);
});
for (const test of await suite.tests()) {
if (testToSuite.has(test))
continue;
testToSuite.set(test, testSuite);
/** @type {Set<Test>} */ (suiteToTests.get(testSuite)).add(test);
}
}
});
for (const test of await rootSuite.tests()) {
const suite = /** @type {import('jest-runner').Test} */(testToSuite.get(test));
if (!startedSuites.has(suite)) {
startedSuites.add(suite);
onStart(suite);
}
const suiteResults = resultsForSuite.get(suite);
// Get results for all browsers
const results = await Promise.all(browsers.map(browser => this._runTest(browser, test)))
results.map(result => {
suiteResults.push(result);
const suiteTests = /** @type {Set<Test>} */ (suiteToTests.get(suite));
if (suiteTests.size === suiteResults.length)
onResult(suite, makeSuiteResult(suiteResults, this._globalConfig.rootDir, suite.path));
})
}
purgeRequireCache(testSuites.map(suite => suite.path));
// Close all browsers
await Promise.all(browsers.map(browser => browser.close()));
}
Also it should be able to configure devices
with browsers
So these are just quick thoughts on the benefits we get from Cypress, and where Playwright might be able to bridge the gap. Some of these might be things better supported on Playwright itself and not the runner (like proxy config).
These are things that Cypress has that I'm not sure Playwright does yet (correct me if I'm wrong):
A headed in-browser test runner UI - developers can watch and debug tests in real time, this is also hooked up to a "watch" mode that picks up code changes. This may be the largest benefit we receive from Cypress in terms of developer experience. The UI test runner allows you to interact with the DOM and pause the test on any steps. (video eg). This UI also surfaces the test failure messages in recorded videos.
Config options to run tests against any base URL -- we point Cypress at any host by changing an env variable CYPRESS_BASE_URL
.
On the same note as the previous ^, ability to propagate any environment variables into browser tests. Cypress uses a convention where any env var prefixed with CYPRESS_
can be discovered and used by tests. https://docs.cypress.io/guides/guides/environment-variables.html#Setting
Ability to listen mock and stub routes and API responses https://docs.cypress.io/api/commands/route.html#Syntax
Option to output recorded video of the test runs -- seems like you heard this one already #5
Proxy configuration for in browser tests https://docs.cypress.io/guides/references/proxy-configuration.html#Set-a-proxy-on-Linux-or-macOS
There's probably more that I'm forgetting, but those are the big ones I can think of.
It would be nice to have auto-wait assertions like those in expect-playwright baked in.
https://github.com/playwright-community/expect-playwright
await expect(page).toHaveSelector("#foobar")
await expect(page).toHaveText("#my-element", "MyValue")
const { it, describe, beforeAll, beforeEach, afterAll, afterEach } = require('@playwright/test');
describe('feature foo', () => {
it('is working correctly', async ({ page }) => {
// Test function
// ...
})
});
I get this error
TypeError: describe is not a function
"playwright-cli": "next",
"@playwright/test": "^0.9.10",
"@playwright/test-runner": "^0.9.22"
We need fit
and fdescribe
.
The config pattern like eslintrc.js and jest.config.json makes sense for setup.ts.
throw err;
^
Error: Cannot find module 'pirates'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/pfeldman/code/playwright-cli/node_modules/@playwright/test-runner/out/transform.js:42:30)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
npx test-runner --screenshot-on-failure
This example not work, option not exists.
error: unknown option '--screenshot-on-failure'
error Command failed with exit code 1.
node:
"playwright-cli": "next",
"@playwright/test": "^0.9.10",
"@playwright/test-runner": "^0.9.22"
node -v
v12.18.4
Rerunning the whole test file could be slow, I want to jump straight to the failing test case.
Is it possible to output the test results to a junit format xml file?
Thanks!
describe.each if you keep duplicating the same test suites with different data. describe.each allows you to write the test suite once and pass data in.
https://jestjs.io/docs/en/api#describeeachtablename-fn-timeout
e2e tests are slower, so I'd like to see test cases (or at least their failures) being reported as they run.
It would be nice to take a screenshot from a test and compare it to a golden screenshot.
Right now e2e tests require users to set up their own server to serve their website, and we treat it as a black box. It would be nice if we had a default solution that did something useful. We should also have a way to integrate with their existing server.
If we don't treat the server as a black box, we can do a lot of cool features like:
The integration would be in the form of a function that takes a localhost url and returns the path to the file on disk which it maps to.
For reference, see microsoft/playwright#2031
This piece of code will not work when running inside Jest:
const playwright = require('playwright');
try {
await page.waitForSelector('.foo');
} catch (e) {
if (e instanceof playwright.errors.TimeoutError) {
// Do something if this is a timeout.
}
} else {
console.log(err);
}
}
It is possible that the TimeoutError is a different instance because Jest test cases is run inside a vm context, whereas PlaywrightRunnerE2E runs in Node.js.
This does work:
if (err.name === 'TimeoutError')
Note: Have not tested this with playwright-runner, but just assuming this is a problem here as well since it was with jest-playwright.
I'd like to have a single command to run like npx setup-playwright
that installs all dependencies, adds a jest.config.js, tsconfig.json, github actions, and a sample test.
We should run tests in parallel.
Currently, every spec file requires importing fixtures
to then import it
, expect
.
// Current
import {fixtures} from '@playwright/test';
const {describe, it, beforeEach, expect} = fixtures;
I suggest we export these objects directly, since most new users don't need to know about fixtures.
// Proposed
import {describe, it, beforeEach, expect} from '@playwright/test';
We can re-export it, but it is handy on the top-level as well.
Are there already some thoughts on how the skipping should work?
More a blacklist or a whitelist of browsers?
Jest currently does something like that in their docs, but without adding custom data (for us browsers).
test.skip('it is not snowing', () => {
expect(inchesOfSnow()).toBe(0);
});
describe.skip('my other beverage', () => {
// ... will be skipped
});
Maybe something like that
test.skip(["firefox"], 'it is not snowing', () => {
expect(inchesOfSnow()).toBe(0);
});
describe.skip("firefox", 'my other beverage', () => {
// ... will be skipped
});
(either an Array or browsers type directly)
Also an interesting mode would be to configure, if the multiple browsers should run in parallel or in order. If its in order, then as proposed in #9 a setup hook can be used to e.g. reset the database.
relates to #42
It would be cool to add custom Jest matchers by default to this lib. An example would be expect(page/element).toContainText('some text')
, taking into account somehow the 3 different kind of text matchers text=example one
(case-insensitive, substring matching), text="example 2"
(exact matching), text=/^\\s*Some text$/i
(regex matching).
Either in 3 separate functions, or just one matcher with all variants.
Also an expect(page/element).toContainElement
(or expect(page/element).toContainSelector
which underneath calls waitForSelector
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.