microsoft/playwright-vscode

Playwright Test Visual Studio Code integration

Playwright Test for VS Code

This extension integrates Playwright into your VS Code workflow. Here is what it can do:

• Playwright Test for VS Code
  • Requirements
  • Install Playwright
  • Run tests with a single click
  • Run Multiple Tests
  • Show browsers
  • Pick locators
  • Debug step-by-step, explore locators
  • Tune locators
  • Record new tests
  • Record at cursor

Requirements

This extension works with Playwright version v1.19+ or newer.

If you are looking for the old extension that supported Playwright v1.14+, switch to v0.0.9 of this extension manually. Having said that, we highly recommend using the latest version of [Playwright Test]!

Install Playwright

If you don't have the Playwright NPM package installed in your project, or if you are starting with a new testing project, the "Install Playwright" action from the command panel will help you get started.

Pick the browsers you'd like to use by default, don't worry, you'll be able to change them later to add or configure the browsers used. You can also choose to add a GitHub Action so that you can easily run tests on Continuous Integration on every pull request or push.

The extension automatically detects if you have Playwright installed and loads the browsers, known as Playwright projects, into Visual Studio Code. By default it will select the first project as a run profile. Inside the test explorer in VS Code you can change this behavior to run a single test in multiple or different browsers.

Run tests with a single click

Click the green triangle next to the test you want to run. You can also run the test from the testing sidebar by clicking the grey triangle next to the test name.

Run Multiple Tests

You can use the Testing sidebar to run a single test or a group of tests with a single click. While tests are running, the execution line is highlighted. Once the line has completed, the duration of each step of the test is shown.

Show browsers

Check the "show browsers" checkbox to run tests with the browser open so that you can visually see what is happening while your test is running. Click on "close all browsers" to close the browsers.

Pick locators

Click the "pick locator" button and hover over the browser to see the locators available. Clicking an element will store it in the locators box in VS Code. Pressing enter will save it to the clip board so you can easily paste it into your code or press the escape key to cancel.

Debug step-by-step, explore locators

Right click and start breakpoint debugging. Set a breakpoint and hover over a value. When your cursor is on some Playwright action or a locator, the corresponding element (or elements) are highlighted in the browser.

Tune locators

You can edit the source code to fine-tune locators while on a breakpoint. Test out different locators and see them highlighted in the browser.

Record new tests

Record new tests by clicking on the "record tests" button in the testing sidebar. This will open a browser window where you can navigate to a URL and perform actions on the page which will be recorded to a new test file in VS Code.

Record at cursor

This generates actions into the existing test at the current cursor position. You can run the test, position the cursor at the end of the test and continue generating the test.

playwright-vscode's Issues

[Feature] run last executed test again

Would be nice to have a shortcut to rerun the most recently run test again.

Docs: Add section of how to use a custom Playwright Test config file

Overly eager in detecting Spec files

Extension is detecting files that are NOT spec-containing files such as Page Object and/or Config modules.

Expectation is that spec detection should only detect (and show) modules that contains specs.

trouble to start tests with example project

When try to start tests in example project i have this picture:

In Debug mode i have error:

In output i have this:

Playwirhght Test For VSCode v0.0.2

VSCode
Version: 1.59.0 (user setup)
Commit: 379476f0e13988d90fab105c5c19e7abc8b1dea8
Date: 2021-08-04T23:13:12.822Z
Electron: 13.1.7
Chrome: 91.0.4472.124
Node.js: 14.16.0
V8: 9.1.269.36-electron.0
OS: Windows_NT x64 10.0.19042

extension no longer starts with vscode as of 0.1.0

the tests tab is not present

there are no run icons on the tests

and attempting to run any playwright commands from the F1 menu results in the following error:

Electron failed to launch

When attempting to run tests using Electron 17.0.1 it appears that playwright is unable to start Electron returning a "electron.launch: Process failed to launch" error message. When I run the tests via the command line using npx playwright test Electron loads up as expected.

VSCode: 1.64.2
Extension Version: 0.1.0
NodeJS: v14.18.3
NPM: 6.14.16
Playwright: 1.19.1
Playwright/Test: 1.19.1
OS: Windows 10 21H1

Refactoring suggestions

Here is the high level code review:

Show stoppers

• Test run should be selectable and should show test output for given run
• There should be a way to start tests from the regular file tree, not from test results
• There should be a one-click action to re-start last tests

UX

• Current test tree UX forces us to discover tests via 'Testing' perspective. I don't think users would do that, they would rather run tests off folders in main tree or from source (play button). We could experiment with only showing results in that perspective.

Bugs

• Only create one testController, not test controller per project
• It looks like persist: false is not respected
• Keep folder / file / suite tree items canResolveChildren = true so that UI did not expand them on load, but when resolveHandler is called, don't run playwright - you already have all the information.
• Spawn npx instead of node + cli.js, explicitly use workspaceFolder for cwd. Don't run Node, don't assume cwd, that way we know we use workspace node and playwright.
• Use vscode.workspace.findFiles('playwright.config.*') to locate configs, after that map configs to vscode.workspace.workspaceFolders, that's how you know where to run npx for these configs.
• Pressing a button should be same as npx playwright filter. I.e. parallel, bulk, not one-by-one. You would need to create your own reporter --reporter=vsCodeExtensionReport.js.

Allow activation if the test-files are not parsable

Currently if there are files in the project which lead to a SyntaxError, the extension does not activate, it does crash.

Context: https://playwright.slack.com/archives/C02CE8Y0W01/p1629743510012100?thread_ts=1629743086.011400&cid=C02CE8Y0W01

no option to stop a test while it's running

when you click the run button in the gutter, there's no way to stop/kill the test

Extension does not appear in the UI

Extension version: v0.0.9
Playwright version: 1.18.0
VS Code: 1.63.2

One package.json in the root. Tests are in root/tests folder named with pattern of *.ts
The extension does not appear in the UI. Is there any way to solve this?

[internal] support env vars

inside the config:

				"playwright.env": {
					"type": "object",
					"description": "Object with environment variables that will be added to the Playwright Test process.",
					"additionalProperties": {
						"type": [
							"string"
						]
					}
				}

[Bug] Extension does not work if something gets logged inside playwright.config

Automatically enable the extension

if a user installs @playwright/test or if the users adds a playwright.config.ts

[l10n] Localize Playwright extension for VS Code to support German

Today Playwright VS Code extension is localized for English, French and Chinese. The German localization is missing.

Display multiple errors from TestInfo.errors

For soft assertions.

Playwright Test for VSCode only reads environment variables on installation

Steps to reproduce:

• Install Playwright for VSCode
• Add environment variables
  "playwright.env": { "baseUrl": "<url here>", "proxyUser": "<username here>", "proxyPass": "password here" },

expected result:
Playwright for VsCode will use the environment variables in the following test processes

Observed result:
Playwright for VsCode does not use the environment variables in the following test processes.

It seems Playwright for VsCode only loads the environment variables when you install the extension. If you have the extension and you change a variable, you'll have to reinstall the extension in order to get it picked up.

Set environment variables

Currently its not possible to set environment variables, it should be possible via the config - the best an existing config entry.

From: https://playwright.slack.com/archives/C02CE8Y0W01/p1629796770022800?thread_ts=1629760890.021100&cid=C02CE8Y0W01

Multi folder workspace support

My workspace has a bunch of folders. Would love to see playwright support it.

Here's what I get today:

azsdke2e
azsdke2e2

BUG: No tests have been found in this workspace - for multi folder workspace

When the workspace includes multiple folders/projects the extension will not find the tests.

Only when the project (where tests are) is opened on it's own tests are discovered

Add Playwright Test version validation

Requirements:

• VSCode 1.59+ (enforced via the extension manifest)
• Playwright Test 1.14.0+ (to be confirmed - we should add verification)
• Operating system (mac/Windows/Linux) should all work and is tested
• Test-file names for activation: using the *.spec.{js,ts}, *.test.{js,ts} naming, we might should make this more flexible?

Can't figure out how to see why a test failed

I have a failing test.

I only see the red X

I do not see the details as to why it failed.

I right clicked on all the places I could find.

I checked the outputs...

I was expecting it in the Playwright Test output window.

azsdke2e
azsdke2e2

Bug: Tests not discovered

Context:

• Playwright Version: 1.19.0-alpha-feb-4-2022
• vsix built from b1ea5d3
• system:
  

Repro:

1. Extract does-not-work.zip, then open in VS Code
2. npm i

Result:
No tests found

Error

Using the same steps with works.zip discovers tests.

Could the More Filters button have an option to show skipped tests?

If possible, could the More Filters button have an option to show skipped tests please?

That is, tests marked as so:
test.skip('this is a test', async....)

Test does get shown twice

Repo: https://github.com/jongio/memealyzer/blob/main/tests/memealyzer.spec.ts

Specifying the `webServer` in the config seems to partially initiate it

I have the webServer configuration specified in my playwright.config.ts file which is setup to run my local webpack dev server. I'm noticing that when my webpack dev server has open: true set, opening the project or expanding a group of Playwright tests will open the URL in my browser. This leads me to believe that the extension is partially initiating the web server when listing tests. This also seems to negatively impact performance of the extension due to it doing something with the server.

Config in subfolder places test results in the wrong location

I'm not sure exactly if this is an extension or playwright core issue, but I'm noticing that when running with VS Code, the test-results folder is placed at the root of my repository but when running with the CLI it is placed in my e2efolder. My project is setup with e2e/playwright.config.ts which I have configured for VS Code like so:

{
  "playwright.configs": ["e2e/playwright.config.ts"]
}

When running with the CLI, it places the test results in the right place when I'm in the e2e directory or if I run yarn workspace e2e test. This leads me to believe it is probably the extension.

My playwright.config.ts file contents:

import { expect, PlaywrightTestConfig } from '@playwright/test'

const config: PlaywrightTestConfig = {
  forbidOnly: !!process.env.CI,
  projects: [
    { name: 'A11y', testDir: 'spec/a11y' },
    // three others that look the same
  ],
  retries: process.env.CI ? 2 : 0,
  use: {
    screenshot: 'only-on-failure',
    trace: 'on-first-retry',
  },
}

export default config

Running the tests does not enter running state immediately

Takes a second to get into the running state on my machine.

Make it work for parameterized tests

I'm using a for loop to iterate over an array with test data to run the same test multiple times and it's detected properly by the test runner extension. But starting the test run it exponentially - for every test that is detected in the loop it runs all the test in the loop

Add Playwright Inspector integration

VSCode has two types of tests:
a) normal tests
-> there we want the inspector if the user enables it
b) debug tests
-> there we don't want the inspector, DEBUG=pw:api should be enough
-> because the VSCode debugger is attached

[Bug] v2 doesn't discover tests in nested folders

Extension activates, but it doesn't discover any tests. My workspace with dir one level deep (tests/test.spec.ts) works, my workspace with dir 2 levels deep does not (ToDoMVC/tests/test.spec.ts)

Monorepo unable to discover playwright tests

I have a monorepo with .net core backend, react spa

├── backend
│   ├── ...
├── frontend
│   ├── e2e
│   │   ├── tests
|   |   |    ├── test.spec.ts
|   |   |    ├── ..
|   |   ├──..
│   │   ├── ..
│   ├── ..
└── .gitignore

2022-02-16T02:53:23.200Z Error: Could not locate Playwright Test. Is it installed? 'npm install -D @playwright/test'

I open workspace @ containing folder of backend & frontend, when i use test explorer, i can only see my backend tests

Make setup without a config work / support step by step setup

repro:

1. mkdir foo
2. cd foo
3. code foo
4. npm init -y
5. npm install @playwright/test@next
6. create a test

expected: works and is activated
actual: not activated

Add color to error output

Not sure if this is a VS Code issue or an issue with this extension, but having colors in the error output would be nice.

Current test results:

While the CLI produces a nicer output:

Path should be regex escaped

The help states that arguments will be used to filter test files, and "each argument is treated as a regular expression". At the moment the path is passed as argument, which (on Windows at least) is not a valid regular expression. This results in

=================
 no tests found.
=================

I think the path passed on the lines below (line 109 specific) should be escaped to be regular expression safe. In my case, if I replace all backslashes with double backslashes, the test is executed correctly.

playwright-vscode/src/playwrightTest.ts

Lines 106 to 110 in 959031d

	const args = [
	...this._buildBaseArgs(config, projectName),
	'--reporter=list',
	`${path}:${line}`
	];

I think the same goes for

playwright-vscode/src/playwrightTest.ts

Line 62 in 959031d

const proc = await this._executePlaywrightTestCommand(config, projectName, [`${path}:${line}`], {

Finally, on line 47 a shortcut is used replaceAll('\\', '\\\\'), but I think this should be a method which escapes all regex characters instead.

playwright-vscode/src/playwrightTest.ts

Line 47 in 959031d

const proc = await this._executePlaywrightTestCommand(config, projectName, ['--list', fileOrFolder.replaceAll('\\', '\\\\')]);

Do not warn that Playwright Test could not be found

Issue: Currently it does warn on every project that it can't be found when Playwright Test is not installed
Solution: Silently log it somewhere but no warning toast

[BUG] VS Code extension not working

Context:

• Playwright Version: 1.15.2
• Extension version: v0.0.7
  

Describe the bug
Test explorer does not appear when viewing test files with extension *.spec.ts

Output window log

cancelling the recorder should delete the generated spec file as well

when you hit F1 and run the "Record Playwright Test" task, it creates this file:

when you click this "Cancel" button, it doesn't save any recorded steps, so it should automatically delete the file

debugging doesn't work

when attempting to debug, it just outputs this to the debug console and does nothing:

main.spec.ts

import { expect, test } from '@playwright/test'

test('basic test', async ({ page }) => {
    await page.goto('https://google.com/')
    const title = page.locator('.navbar__inner .navbar__title')
    await expect(title).toHaveText('Playwright')
})

playwright.config.ts

import { PlaywrightTestConfig } from '@playwright/test'

const config: PlaywrightTestConfig = {
    projects: [
        {
            name: 'Chrome Stable',
            use: {
                browserName: 'chromium',
                // Test against Chrome Stable channel.
                channel: 'chrome',
            },
        },
    ],
}
export default config

playwright test output

2021-09-21T05:07:05.236Z Executing command: node_modules\@playwright\test\lib\cli\cli.js test --retries=0 --reporter=json --list \. 
2021-09-21T05:07:10.458Z Creating test controller: Playwright TestChrome Stable 
2021-09-21T05:07:10.459Z Executing command: node_modules\@playwright\test\lib\cli\cli.js test --retries=0 --project=Chrome Stable --reporter=json --list c:\\Users\\user\\playwright-demo\\tests\\main\.spec\.ts 
2021-09-21T05:07:12.926Z TestFile._updateFromDisk default and Chrome Stable 
2021-09-21T05:07:23.871Z Running test basic test [Chrome Stable] debug=true 

Split tests in the tree based on their folder structure

At the moment all test files are shown as a list in the root of the tree, but this results in many similar entries if you have a bunch of files in the same folder.
I think it would make much more sense if the tests were structured by their folder structure. That way it would be much easier to find a specific test, and it would be very easy to run all tests in a specific folder.
I believe this how the jest vscode extension handles it as well.

[Bug] v2 doesn't work with nested playwright-core module

npm 6 installed playwright-core module inside @playwright/test (thinks there is a conflict for some reason?), as a result, extension is not activated. Upgrading npm to 7.24.2 adds playwright-core to root, fixing the issue.

command 'python.refreshTests' not found

When I try to refresh tests (would be great if the list auto-refreshes btw), I get this error:

command 'python.refreshTests' not found

How can I fix this?

Don't add every open JavaScript file to the Test Explorer

Currently every JS/TS file which gets opened gets added to the Test Explorer. Some validation should be necessary to filter out non-test files.

Code pointer: updateNodeForDocument function.

Test tree isn't updating automatically

I'm creating new tests but the test tree isn't updating.
I need to reload VS Code in order to see the newly created tests.

Adopt status bar name

If you give the status bar item a name it will have a nice label here:

I think without a name hiding this will also hide all status bar entries for the extension.

[Question] Is it possible to print the test results in the terminal?

First of all, I'd like to thank you for the brilliant plugin

Now the output of the test results in the terminal is just simple text of Running and Completed spec files.

But, as a user, I'd like to see the test results in the terminal in the selected reporter style (e.g dot or list ), which I selected in the configuration file.

Actual:

Expected:

Does not work with pnpm workspace

I have a root pnpm workspace that includes a folder ./e2e-tests with a default Playwright setup.
The test runner says : "No tests have been found in this workspace yet."
I tried to set dotnet-test-explorer.testProjectPath with no luck.

Using pnpm in a non-workspace setup works fine,

All Playwrite setups are done with the default npm init playwright new-project command.

Any ideas how to fix that?

Add a changelog

You're using GH releases right now but the way to get the changelog integrated into VS Code and the marketplace is by adding a CHANGELOG.md file to the root of the extension folder.

Enable Playwright Debug take a lot of space/is distracting

Not sure this warrants its own status bar entry? I know you can hide this easily but most users don't do this so a large portion of the status bar is taken up by this entry.

if multiple soft assertions fail, only first one is shown! :'(

My test:

test('should verify math', async ({ }) => {
  expect.soft(1, 'math should work!').toBe(2);
  expect.soft(2, 'math should work - 2!').toBe(3);
});

Detect Playwright configs in subfolders out of the box

Many of the projects I work on have Playwright tests in a dedicated subdirectory (e.g. e2e) separate from the main project which might be in app or src. The extension by default only looks for a playwright.config.ts file in the root so this requires that I set e2e/playwright.config.ts in my settings to make the tests discoverable.

This issue could of course also apply for monorepos with multiple test projects, but this is probably a first step towards that if others have a use case for multiple Playwright projects in a single repo.