percy/percy-puppeteer

Visual testing with Puppeteer and Percy

@percy/puppeteer

Percy visual testing for Google Puppeteer.

Installation

$ npm install --save-dev @percy/cli @percy/puppeteer

Usage

This is an example using the percySnapshot function. For other examples of puppeteer usage, see the Puppeteer docs.

const puppeteer = require('puppeteer');
const percySnapshot = require('@percy/puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('http://example.com/', { waitUntil: 'networkidle2' });
  await percySnapshot(page, 'Example Site');

  await browser.close();
})();

Running the code above directly will result in the following logs:

$ node script.js
[percy] Percy is not running, disabling snapshots

When running with percy exec, and your project's PERCY_TOKEN, a new Percy build will be created and snapshots will be uploaded to your project.

$ export PERCY_TOKEN=[your-project-token]
$ percy exec -- node script.js
[percy] Percy has started!
[percy] Created build #1: https://percy.io/[your-project]
[percy] Running "node script.js"
[percy] Snapshot taken "Example Site"
[percy] Stopping percy...
[percy] Finalized build #1: https://percy.io/[your-project]
[percy] Done!

Configuration

percySnapshot(page, name[, options])

• page (required) - A puppeteer page instance
• name (required) - The snapshot name; must be unique to each snapshot
• options - See per-snapshot configuration options

Upgrading

Automatically with @percy/migrate

We built a tool to help automate migrating to the new CLI toolchain! Migrating can be done by running the following commands and following the prompts:

$ npx @percy/migrate
? Are you currently using @percy/puppeteer? Yes
? Install @percy/cli (required to run percy)? Yes
? Migrate Percy config file? Yes
? Upgrade SDK to @percy/[email protected]? Yes

This will automatically run the changes described below for you.

Manually

Import change

In v1.x there wasn't a default export of the package (only a named export). With v2.x the named export is removed and there is only a default export.

// old
import { percySnapshot } from '@percy/puppeteer';
const { percySnapshot } = require('@percy/puppeteer');

// new
import percySnapshot from '@percy/puppeteer';
const percySnapshot = require('@percy/puppeteer');

Migrating Config

If you have a previous Percy configuration file, migrate it to the newest version with the config:migrate command:

$ percy config:migrate

percy-puppeteer's Issues

`@percy/core` development dependency issue

The problem

I am working on a common library for our 12 projects setup.
There is an error compiling TypeScript due to missing @percy/core dependency.

Environment

• Node version: 16.15.0
• @percy/cli version: N/A
• Version of Percy SDK you’re using: N/A
• OS version: Ubuntu 20 LTS
• Type of shell command-line [interface]: bash

Details

@percy/puppeteer has @percy/core listed as a development dependency, but should be also an optional peer dependency for TS users.

Debug logs

$ tsc --build
node_modules/@percy/puppeteer/types/index.d.ts:2:33 - error TS2307: Cannot find module '@percy/core' or its corresponding type declarations.

import { SnapshotOptions } from '@percy/core';

Code to reproduce issue

Package manifest:

{
  "name": "@xxx/e2e-test-utils",
  "description": "XXX generic end-to-end test utilities.",
  ...
  "dependencies": {
    "@percy/puppeteer": "^2.0.2"
  },
  "devDependencies": {},
  "peerDependencies": {
    "puppeteer": "^14.4.1"
  },
  ...
}

Helper function:

import percySnapshot from '@percy/puppeteer';

export const takeSnapshot = (name: string) => percySnapshot(page, name);

Error: Caught error after test environment was torn down

Im having problems getting the latest version of percy to work with puppeteer. Before this new CLI update it worked very well.

The problem seems to be that i run 23 x 3 of these tests:

  it('should not have visual regressions in new dark theme (percy)', async () => {
    const page = await browser.newPage();
    await page.setBypassCSP(true);
    await page.goto(url, { waitUntil: 'load' });
    await page.evaluate(() => {
      document.querySelector('ids-theme-switcher').setAttribute('mode', 'dark');
    });
    await percySnapshot(page, 'ids-textarea-new-dark');
  });

And at a certain point it will stop and error with

    : Timeout - Async callback was not invoked within the 25000 ms timeout specified by jest.setTimeout.Timeout - Async callback was not invoked within the 25000 ms timeout specified by jest.setTimeout.Error:

      21 |   });
      22 |
    > 23 |   it('should not have visual regressions in new contrast theme (percy)', async () => {
         |   ^
      24 |     const page = await browser.newPage();
      25 |     await page.setBypassCSP(true);
      26 |     await page.goto(url, { waitUntil: 'load' });

      at new Spec (node_modules/jest-jasmine2/build/jasmine/Spec.js:116:22)
      at Suite.<anonymous> (test/ids-text/ids-text-percy-test.js:23:3)

(node:20849) UnhandledPromiseRejectionWarning: Error: Caught error after test environment was torn down

Protocol error (Page.navigate): Target closed.
    at /Users/tmcconechy/Dev/enterprise-wc/node_modules/puppeteer/src/common/Connection.ts:270:57
    at new Promise (<anonymous>)
    at CDPSession.send (/Users/tmcconechy/Dev/enterprise-wc/node_modules/puppeteer/src/common/Connection.ts:269:12)

Seems like either my test is wrong but i cant see anything else i can do? Or there is some issue with percy that makes it "crash out" after a certain number of screenshots.

This is the repo/build check if this helps https://github.com/infor-design/enterprise-wc/pull/59/checks?check_run_id=2308847368#step:8:1512
Any suggestions?

Capturing a single frame inside a page

The request is to expand API to do percySnapshot(frame: Frame), not just a Page. It's possible that the current API works even now, since Frame and Page APIs are interchangeable for the most part. But would be good to have a typed and confirmed support.

More helpful error logging during script injection

It would be really helpful to have better error logging here:

percy-puppeteer/lib/index.ts

Lines 28 to 37 in ace51ad

	try {
	await page.addScriptTag({
	path: agentJsFilename()
	})
	} catch (err) {
	// Certain CSP settings prevent Puppeteer from injecting scripts. See:
	// https://github.com/GoogleChrome/puppeteer/issues/2644
	console.log(`[percy] Could not take snapshot named '${name}', maybe due to stringent CSPs. Try page.setBypassCSP(true).`)
	return
	}

I get the CSP error pretty regularly when setting up stuff with Percyscript, and CSP usually isn't the problem. Right now, I have a setup that works locally but fails in CI, and it would be extremely helpful to know what is actually throwing the error here.

I also got this error earlier in the setup, but again it wasn't a CSP issue. I simply didn't have my Percy token loaded into the env properly. It probably took me 5x as long to debug because of the misleading and unhelpful message.

During snapshot page is reloaded causing test fail

I started to integrate Percy into my puppeteer testing flow and came across a failure I can't seem to work around. Percy successfully gets a Dom snapshot but has a final snapshot step which invokes page.get reloading test harness.

When the test harness reloads it aborts the current test and Percy fails.

I looked through the code and I can't figure out why we need to reload the page given it was already loaded. Perhaps to reset the injected Percy code?

One solution that just came to me is stubbing the page object to ignore the extra reload but curious if there are other options.

Capturing a selector instead of a page?

Hey folks!

We're evaluating how to best do Visual Regression tests for a range of problems. The first thing we're trying to tackle is our design system. Now, we don't want to use Storybook (we're using docz), but I thought I would be able to relatively easily use puppeteer to create snapshots of marked elements on our page. I have everything working so far, except I don't have a way to target sub-elements on a page instead of the whole page (the percySnapshot function takes a page as argument, which won't even run because typescript fails the test). Is this something you could help me with? I haven't dug into either puppeteer or these bindings enough yet, but think I could contribute it. I guess my first question would be; is this an option, an extra argument / overload, or an entirely new method?

Thanks!

percySnapshot sometimes captures part of the page and sometimes full page

Using Percy with Puppeteer, for example with simple script:

await page.goto(url); await percySnapshot(page, 'name');

I receive sometimes a screenshot of the full page and sometimes only part of it.
It happens pretty often and was not able to find a consistent solution.

Not sure is it a Percy's issue or Puppeteer itself.

Improve error message when large DOM is sent

When a DOM is between 10mb and 15mb an unclear error message appears. Add better handling around this case by outputting a clear error message.

`render.percy.local` error

Hi, I'm running into an issue using @percy/cli with @percy/puppeteer and some quick googling didn't seem to turn up anything particularly relevant. Specifically I'm trying to take snapshots at various points during my company's initial onboarding flow. At one point this was working as expected but I now see the following error for every screenshot in my build:

When I run the test without headless mode it steps through each page and they look like they should. I don't see anything particularly suspicious in the debug logs either. Anyone have any suggestions?

Debug logs (with skipped remote resources omitted)

``` $ jest --runInBand src/apps/reception/__tests__/happy_path.test.ts Determining test suites to run...[percy:config] Found config file: package.json (0ms) [percy:config] Using config: { discovery: { allowedHostnames: [ 'odfil.es', 'odfil.es' ] }, snapshot: { enableJavaScript: true }, version: 2 } (13ms) [percy:core] Percy has started! (1457ms) [percy:core] Created build #591: https://percy.io/3993b1f6/reception/builds/9391952 (1458ms) [percy:core] --------- (12949ms) [percy:core] Handling snapshot: (12949ms) [percy:core] -> name: SIMPLE_RADIO_BUTTON (12949ms) [percy:core] -> url: https://consumer.simplersell.com/seller-lead/615c110b-1916-4f5c-839c-54b50dfcc9c0/relationship (12949ms) [percy:core] -> widths: 414px, 768px, 1400px (12949ms) [percy:core] -> clientInfo: @percy/puppeteer/2.0.0 (12949ms) [percy:core] -> environmentInfo: puppeteer/2.1.1 (12949ms) [percy:core] -> requestHeaders: {} (12949ms) [percy:core] -> authorization: undefined (12949ms) [percy:core] -> domSnapshot: <script async="" src="https://s.pinimg.com/ct/lib/main.dec9de31.js"></script><script type="text/javascript" async="" src="https://d2hrivdxn8ekm8.cloudfront.net/tag-manager/63fabb82-e541-4265-926d-1f4bfbfe9aaa-additional-latest.js"></script><script async="" src="//acdn.adnxs.com/dmp/up/pixie.js"></script><title>Opendoor</title><style data-emotion=""></style><style data-emotion="css"></style><style data-emotion="css"></style>

CSS viewport units are not correct on chrome headless

When using vw or vh units, it's messing up with Chrome's screenshots.

Expected Result (Firefox):

https://storage.googleapis.com/percy-images/71e5fe95419f5eefdd8fe2ca59c0adf29e476804d325f77f29cf447ef1d6b761

What we've got (Chrome):

https://storage.googleapis.com/percy-images/084d15a107e3d710529c47b47632700e8d00acfa767aba2cddb1688d3a907d16

Seems related to puppeteer/puppeteer#703

Is there any way of fixing that issue in percy?

Percy agent communication blocked by CSP

Whilst trying to add Percy in https://github.com/buildkite/site/compare/add-percy I hit the following error:

(node:72) UnhandledPromiseRejectionWarning: AssertionError [ERR_ASSERTION]: console.error not allowed in production: Refused to connect to 'http://localhost:5338/percy/healthcheck' because it violates the following Content Security Policy directive: "connect-src 'self' https://www.google-analytics.com".

The CSP being used is:

default-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline' https://www.google-analytics.com; img-src 'self' 'unsafe-inline' data: https://www.google-analytics.com; connect-src 'self' https://www.google-analytics.com; media-src 'self' https://d3lj8s78qytm30.cloudfront.net

I could use Puppeteer’s page.setBypassCSP and disable CSP entirely, but because it's black-box testing of the prod Docker image (and we want to test the CSP doesn't break anything in the Percy screenshots) I'd prefer to leave it as-is.

Is there any known workaround for this, with the new percy/agent setup? I searched the Percy docs for CSP and Content-Security-Policy, but I couldn't find anything.

Local fonts not loaded on percy rendering environment

Hi,

Faced an issue with local fonts failing to be rendered on percy rendering environment.
It works fine locally, it was verified by additional listener for this font loading.

Example of expected appearance:

Example of received screenshot from percy:

Tried:

• increasing timeouts
• adding allowed-hosts from https://docs.percy.io/docs/font-services

Issue is tricky for me and is not reproducible for 100%.

Please advise how to fix or workaround it on my side, or will it be fixed on percy side(if so, when?).

Thank you in advance.

Unfortunately, spent around 15% of monthly screenshots trying to fix this issue until realised it cannot be fixed on my side.

Support for Jest / use of ES modules dynamic import in SDK

Apologies if this isn’t the best place to report this! I couldn’t find information about Jest support anywhere in official resources so I’m not sure whether this would be considered a bug or not.

The problem

In #411, the import of SDK utils has been switched to use an ES modules dynamic import. This is problematic for users of Jest, because Jest uses Node’s VM API to execute code. The VM modules API for ES modules support is experimental (only available behind the --experimental-vm-modules feature flag).

For the Puppeteer SDK – this means v2.0.0 works fine in Jest, and v2.0.1 doesn’t work, with the following error when running percySnapshot:

TypeError: _vm(...).SyntheticModule is not a constructor

Environment

• Node version: v16.13.2
• @percy/cli version: v1.1.2
• Version of Percy SDK you’re using: @percy/[email protected]
• If needed, a build or snapshot ID:
• OS version: macOS Monterey 12.3
• Type of shell command-line [interface]: zsh?

Details

Jest has experimental support for ES modules, so this can be resolved by adding the correct feature flag for Node to expose its vm.Module API. I believe there are a few things worth addressing in the SDK nonetheless (assuming the SDK attempts to support Jest):

• There are many known issues with this experimental API in Node (hence why it’s experimental, and why Jest documents it as such), so at least as of now even with the latest Node 18 release, it won’t be as reliable as the equivalent CommonJS implementation.
• This change shouldn’t be in a patch release since it’s a breaking change for Jest users.
• If the SDK keeps using ES modules anyway, then this at least needs to be documented clearly.

Debug logs

The only relevant bit is the stack trace.

[…]
[percy:cli:exec] Running "jest" (1ms)
  ● Groups › loads

    TypeError: _vm(...).SyntheticModule is not a constructor

      39 |   beforeAll(async () => {
      40 |     await page.goto(`${TEST_ORIGIN}/admin/groups/2/`);
    > 41 |     await percySnapshot(page, 'test name')
         |           ^
      42 |   }, 10000);

      at Runtime.loadCjsAsEsm (../../../node_modules/jest-runtime/build/index.js:657:20)
      at percySnapshot (node_modules/@percy/puppeteer/index.js:9:15)
      at Object.<anonymous> (groups.test.js:41:11)

A worker process has failed to exit gracefully and has been force exited. This is likely caused by tests leaking due to improper teardown. Try running with --detectOpenHandles to find leaks.

[…]
Ran all test suites.
[percy:core:browser] Closing browser (25951ms)
[percy:core:browser] Browser closed (29ms)
[percy:core] Build not created (0ms)

Code to reproduce issue

I can provide a small reproduction case once confirmed Jest is indeed supported.

---

Workarounds

For people running into this, there are three possible workarounds:

Revert to v2.0.0 of the Puppeteer SDK

It works.

Run the v2.0.1 SDK with Node flag

Following Jest’s documentation. Here is what it looks like:

NODE_OPTIONS=--experimental-vm-modules percy exec -- jest

This will output a lot of warning messages.

Use @percy/sdk-utils directly

The code of v2.0.0 of the SDK can be copy-pasted and tweaked as needed to improve Jest support.

`ReferenceError: window is not defined` with Jest 28 and jest-puppeteer

The problem

ReferenceError: window is not defined is thrown after upgrading to Jest 28.
Same setup works fine with Jest 27 though.

Environment

• Node version: 16.15
• @percy/cli version: 1.6.0
• OS version: Ubuntu 20 LTS
• Type of shell command-line [interface]: bash

"dependencies": {
    "@percy/cli": "^1.6.0",
    "@percy/puppeteer": "^2.0.2",
    "expect-puppeteer": "^6.1.1",
    "jest": "^28.1.2",
    "jest-puppeteer": "^6.1.1",
    "puppeteer": "^15.3.2"
},

Details

ReferenceError: window is not defined error occurs when referencing percySnapshot method after upgrading to Jest 28.
Installing JSDOM environment explicitly does not help either. https://jestjs.io/docs/upgrading-to-jest28#jsdom

Debug logs

Test suite failed to run

    The error below may be caused by using the wrong test environment, see https://jestjs.io/docs/configuration#testenvironment-string.
    Consider using the "jsdom" test environment.
    
    ReferenceError: window is not defined

      at Object.<anonymous> (node_modules/@percy/sdk-utils/dist/bundle.js:347:9)
      at Object.<anonymous> (node_modules/@percy/puppeteer/index.js:1:35)

Code to reproduce issue

Install Jest 28 and configure latest jest-puppeteer as per documentation https://jestjs.io/docs/28.0/puppeteer#use-jest-puppeteer-preset
Run simple test:

import percySnapshot from '@percy/puppeteer';

describe('E2E test', () => {
  it('renders', async () => {
    await page.goto(`https://github.com`);

    await percySnapshot(page, 'homepage');
  });
});

PNG images are not displayed

PNG images are not loaded in screenshot results (on this page SVG - displayed, JPG - random (sometime show), PNG - only alt text): https://percy.io/10ecbde0/Lition/builds/8869567/changed/503202803?browser=firefox&browser_ids=9%2C14&subcategories=approved&viewLayout=side-by-side&viewMode=new&width=1920&widths=375%2C1920
On puppeteer screenshot displayed all images.

This happened in period from 2021-02-08 - 2021-02-15, before - everything works fine

Snapshot requires that the page URL must be HTTP or HTTPS

Hi im having an issue were percy service, it throws an error if I try to snapshot a URL that is a file I opened from the system.

The currently flow is:

1. user client sends an HTML to the backend
2. backend parses the HTML and get what ever data it needs
3. generates a new HTML
4. puppeteer sends back a PDF

I had to do a work around this simple scenario and create a ExpressJS server to provide that generated HTML as http://localhost/...

My code looks like

  const filePath = resolve(fileName);

  await page.goto(`file://${filePath}`, {
    waitUntil: "networkidle0"
  });

but when I run my tests/snapshot on that code I get

Is there a way to handle this?

executing percySnapshot on multiple pages

Hey there!

I have a really long list of tests (~1000) and I wouldn't want to run them synchronously, it would take a really long time.

That's why I'm creating multiple pages on the same browser instance (this is different from the example repo in that the browser object is recreated for each test, it probably doesn't matter because I tried creating a new browser instance for each async operation).

Unfortunately, whenever I try to run the snapshots for multiple pages I get this kind of error:

UnhandledPromiseRejectionWarning: Error: Request is already handled!