Feature request: Automatically take screenshot when test fails

[argos-ci]

Feature request: Automatically take screenshot when test fails

[redrockzee]

Use this package:
https://github.com/jeeyah/jestscreenshot

[mejackreed]

I like the idea of the automation behind this, and +1 to uploading to S3 for a CI debug. But as a user of jest-puppeteer, it would feel a bit of a reach to me to have this on by default and not configurable. I agree this would be a great feature to have but I could see an issue where for someone breaking a build, many unexpected images are now created which could slow down the testing process. I love the idea of this 👏 @melissachang !

So +1 to opt-in and the ability to have it configurable.

[callumlocke]

Is anyone else working on this? If not I might try and get something working...

[amid2887]

This is a very desirable feature. As far as I understand, this should be done with the help of a jasmine reporter. But the version of jasmine that is used only works with synchronous calls. And this creates some problems.
At the moment, the best I've been able to do for myself is to redefine the "it" calls. This method has some disadvantages, but it may be useful to someone.

// package.json
{
  // jest section
  "preset": "jest-puppeteer",
  "setupTestFrameworkScriptFile": "./jest-setup.js"
}

// jest-setup.js
const fs = require('fs');
const jestPreset = require('jest-puppeteer/jest-preset');
const expectPuppeteer = require(jestPreset.setupTestFrameworkScriptFile);

const { browser } = global;

const env = jasmine.getEnv();
// create a reference to original "it" method
const { it: originalIt } = env;

// replace jasmine "it" method with the new one which returns original
env.it = (name, fn, ...args) => {
  const newFn = async (...fnArgs) => {
    try {
      return await fn(...fnArgs);
    } catch (e) {
      // get the filename where the error occurred
      // PS this is a weak place and it does not work for 100% - for some errors can be "null"
      let message = e.stack;
      message = message.match(new RegExp(`${__dirname}(.*)`));
      if (message && message[1]) {
        message = message[1]
          .replace(/^.*[\\/]/, '')
          .replace(/:.+$/, '');
      }

      // run through opened pages also take a screenshot and save source code of the page to the "./artifacts" folder
      const pages = await browser.pages();
      for (let page of pages) {
        const title = await page.title();
        // filter "about:blank" pages and pages without the title attribute
        if (title) {
          const fileName = `${message}__${name}__${title}`.replace(/\s+/g, '_');
          const path = `${__dirname}/artifacts/${fileName}.png`;
          await page.screenshot({ path });
          const content = await page.content();
          await fs.writeFile(`${__dirname}/artifacts/${fileName}.html`, content);
        }
      }
      throw e;
    }
  }
  return originalIt(name, newFn, ...args);
}

module.exports = expectPuppeteer;

[wangtao0101]

use this setupTestFrameworkScriptFile in jest config

setupTestFrameworkScriptFile: './hack-expect.js',

// hack-expect.js
const expectPuppeteer = require('expect-puppeteer');
const getPuppeteerType = require('expect-puppeteer/lib/utils').getPuppeteerType;

const puppeteerMatchers = {
  'toClick': 0,
  'toDisplayDialog': 0,
  'toFill': 0,
  'toFillForm': 0,
  'toMatch': 0,
  'toMatchElement': 0,
  'toSelect': 0,
  'toUploadFile': 0,
  'not': {
    'toMatch': 0,
    'toMatchElement': 0,
  },
}

function createMatcher(actual, matcher) {
  return async (...args) => {
    try {
      return await matcher(...args);
    } catch (error) {
      await actual.screenshot({ path: './screenshot.png' });
      throw error;
    }
  }
}

if (typeof global.expect !== 'undefined') {
  const originalExpect = global.expect
  global.expect = (actual, ...args) => {
    const matchers = originalExpect(actual, ...args);

    const type = getPuppeteerType(actual);
    if (type == null) {
      return matchers;
    }
    Object.keys(puppeteerMatchers).forEach(key => {
      if (key === 'not') return;
      if (matchers[key]) {
        matchers[key] = createMatcher(actual, matchers[key])
      }
    })

    Object.keys(puppeteerMatchers.not).forEach(key => {
      if (matchers.not[key]) {
        matchers.not[key] = createMatcher(actual, matchers.not[key])
      }
    })
    return matchers;
  }
  Object.keys(originalExpect).forEach(prop => {
    global.expect[prop] = originalExpect[prop];
  })
}

[gregberge]

Good idea!

[melissachang]

Thanks. My recommendations:

• Take advantage of the fact that this repo is relatively new, and don't make an option for this. Just always do this by default.
• When someone requests an option, you can add an option. But it should be on by default, because I think it would be useful most of the time.

[yanivefraim]

@neoziro @melissachang - I played with this on one of our projects (created a reporter which take screenshots for failing tests). It is not 100% ready but I would be happy to help here.

One nice feature: upload those screenshots (s3?) so we can see those on CI as well.

WDYT?

[melissachang]

Good point, default off is probably safest. CI integration would be awesome.

[gregberge]

@callumlocke you are welcome 😀

[wangtao0101]

@amid2887 if you want to getFilename and testName, the right way is:

const getState = require('expect/build/jest_matchers_object').getState;
console.log(getState());

And await browser.pages() may return unexpected page because of concurrency.

[amid2887]

@wangtao0101 thanks for the advice - it works more stable than previous one =)
I forgot to mention that I use jest --runInBand mode because I have to login in tests with different user types and tests can have a few opened window at the same time.
But yeah, this part should be done in another way...
Does anyone have any suggestions?

[gregberge]

Follow up in #131