testimio/root-cause

🔍 [DEPRECATED] Root Cause is a tool for troubleshooting Puppeteer and Playwright tests. 🔎

[DEPRECATED] - This repository is no longer maintained and should not be used

Testim Root Cause

This is the Testim Root Cause monorepo. Please check out the documentation.

Please see the contributing guide and note this project has a Code Of Conduct

Getting Started

Please check out the getting started guide to integrate with Jest, Mocha, or to run standalone.

Testim Root Cause

Root Cause is a tool for troubleshooting Puppeteer and Playwright tests.

We believe modern automation frameworks like Puppeteer, Playwright and Selenium are pretty fast and useful but maintaining and debugging tests is hard.

Root Cause adds features to simplify root cause analysis of Puppeteer and Playwright test runs. Root Cause captures screenshots, network HAR files, and console logs from each test run and saves them to a local drive. The screenshots highlight the action taken at each step and are easily viewed in succession through an intuitive UI to demonstrate the test flow or to identify where a test failed. Additionally, the console logs are parsed to each test step and network HAR is available to deep-dive into failed steps.

Note that this project is a yarn workspace and packages inside have specific READMEs.

Testim.io

This project is maintained and supported with ❤️ by Testim.io.

License

AGPL.

root-cause's Issues

Root Cause Integration Error when working with jest-playwright

Hello, I'm seeing the following when trying to integrate root-cause into my jest-playwright project. Here's the error I'm getting:

  ● Test suite failed to run

    Root Cause Integration Error

      at Object.ensurePrerequisite (node_modules/@testim/root-cause-jest/lib/helpers.js:20:11)
      at Object.<anonymous> (node_modules/@testim/root-cause-jest/lib/forSetupFilesAfterEnv.js:23:9)

Error: ENOENT: no such file or directory, open 'C:\Users\tyler.jackson\source\repos\SkyKick.Manager\Web\testFrameworks\playwright\.root-cause\runs\1620327124642\runConclusion.json'

My jest.config.js:

'use strict';

const playwrightPreset = require('jest-playwright-preset/jest-preset.json');

const runId = Date.now().toString();

module.exports = {
    ...playwrightPreset,
    testMatch: ["**/__tests__/**/*.+(ts|js)", "**/?(*.)+(spec|test).+(ts|js)"],
    transform: {
        "^.+\\.(ts)$": "ts-jest",
    },
    reporters: [
        ['@testim/root-cause-jest/lib/reporter/default', { runId }],
    ],
    testEnvironmentOptions:{
        browsers: ["chromium", "firefox", "webkit"],
        debugOptions: {
            browsers: [
                'chromium'
            ],
            launchOptions: {
                headless: false,
                devtools: true
            }
        },
    },
    setupFilesAfterEnv: [
        ...playwrightPreset.setupFilesAfterEnv, '@testim/root-cause-jest/lib/forSetupFilesAfterEnv'
    ],
    globals: {
        runId,
    },
};

npm v7.6.3
node v15.12.0

Add after-publish / after-package integration tests

Our current integration tests are running agains the source in the repo/build code,
via workspace and not real npm package (or tarballs)

In the real world, the code will run from inside npm package
It's very possible that something will get wrong in the packaging/publishing process (eg different paths)

We need to create an infra that will let us to run the same integration tests that we have against the to-be-published packages (using tarball or verdaccio local registry)

It's very important to also test the CLI inside each package

An easier way to show "before all" in tests

Users have run into a case where they login inside a beforeAll clause and they can't see those actions in the viewer - would be nice to see them somehow.

Firefox not working for RC

When copying URL for test results into Firefox, page is blank.

I ran test in terminal. Copied link. My default browser is Firefox, so it opened in Fx. Get a blank screen.

Root Cause Integration Error in jest 27

Hi,
After updating jest version to 27+ I keep getting the error:

FAIL ./example.test.js
● Test suite failed to run

Root Cause Integration Error

  at Object.ensurePrerequisite (node_modules/@testim/root-cause-jest/lib/helpers.js:20:11)

Error: ENOENT: no such file or directory, open 'C:\Users\Zordex\RiderProjects\jest-puppeteer-root-cause.root-cause\runs\1628776599422\runConclusion.json'

You can check the following example for quick reference

Showing test artifacts of new tabs

Hey,
I would like to see the steps for new tabs (opened within a part of a test) inside the root-cause/cloud.
Using Playwright:
const [newTab] = await Promise.all([ context.waitForEvent('page'), page.click('some_selector_that_opens_a_new_tab') ])

Performance Reporting

To sum up information and discuss outside the PR.

Useful links:

• https://addyosmani.com/blog/puppeteer-recipes/
• https://github.com/paulirish/automated-chrome-profiling/blob/master/get-cpu-profile.js

Dependency error asking for puppeteer

Running npm install @testim/root-cause on code with only Playwright test is also expecting for Puppeteer dependencies to be installed.
Is there a way to install Root cause for only Playwright?

Config file does not work

Hi,
I've created a config file root-cause-main\integration-examples\another-jest-and-puppeteer\.root-causerc.json with the following code inside (taken from the example):

{
    "features": {
        "screenshots": {
            "format": "png",
            "fullPage": true
        },
        "networkLogs": true,
        "jestAssertions": true,
        "console": false
    }
}

But the saved screenshots are still .jpg
Is there something I'm missing?
Thanks.

• node v14.15.5
• npm 6.14.11

Example for Jest and Playwright for users who don't use jest-playwright library

Firstly, thank you for this awesome tool! I am using a e2e test framework where I have Jest and Playwright installed separately. The jest-playwright example I can see from this repository is using jest-playwright package and jest-playwright-preset. With my set up, I am getting lots of errors while running the example test because of not having lots of global variables what jest-playwright package offers. Here is the error I am getting:

$ jest "./__tests__/sample.tests.jest-playwright.js"

 FAIL  __tests__/sample.tests.jest-playwright.js (10.07 s)
  ● Example root cause for jest and playwright › testim demo example

    **Global page is missing**

      at Object.forBeforeEachOwnGlobals (node_modules/@testim/root-cause-jest/lib/helpers.js:81:15)

  ● Example root cause for jest and playwright › testim demo example

    **TypeError: global.endTest is not a function**

      at Object.forAfterEachEndTestOwnGlobals (node_modules/@testim/root-cause-jest/lib/helpers.js:130:12)

  ● Example root cause for jest and playwright › wikipedia example fail

    **Global page is missing**

      at Object.forBeforeEachOwnGlobals (node_modules/@testim/root-cause-jest/lib/helpers.js:81:15)

  ● Example root cause for jest and playwright › wikipedia example fail

    TypeError: Cannot read property 'getProperty' of null

Could you please provide a simple example of using root cause where users like me use jest as test framework and playwright for browser automation but not using jest-playwright package? Thanks.

Screenshots are not generated when creating a new page before each test (jest-puppeteer)

Hi,
When I launch all the tests on the same page everything works great, but when I try to create a new page for each test, root-cause stops generating screenshots.
I've created a demo example to better describe the issue.

Error creating execution

Hi, I have a bug when a test fails because it exceeds the timeout:

Timeout - Async callback was not invoked within the 60000 ms timeout specified by jest.setTimeout.Error: Timeout - Async callback was not invoked within the 60000 ms timeout specified by jest.setTimeout.

  at mapper (node_modules/@jest/core/node_modules/jest-jasmine2/build/queueRunner.js:27:45)

✖ Error Creating Execution: Error creating execution:Internal Server Error
{"success":false,"errorMessage":"failed to create execution (run result)"}
TestimApiError: Error creating execution:Internal Server Error
{"success":false,"errorMessage":"failed to create execution (run result)"}
at TestimExecutionsApi.createExecution (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@testim/root-cause-jest/node_modules/@testim/root-cause-core/lib/testim-services-api/testimExecutionsApi.js:45:19)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
at RootCauseExecutionsApi.createExecution (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@testim/root-cause-jest/node_modules/@testim/root-cause-core/lib/testim-services-api/rootCauseExecutionsApi.js:49:9)
at Object.persist (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@testim/root-cause-jest/node_modules/@testim/root-cause-core/lib/persist.js:42:34)
at EnhancedDefault.onRunComplete (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@testim/root-cause-jest/lib/reporter/RunConclusion.js:58:13)
at EnhancedDefault.onRunComplete (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@testim/root-cause-jest/lib/reporter/default.js:82:9)
at ReporterDispatcher.onRunComplete (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@jest/core/build/ReporterDispatcher.js:88:9)
at TestScheduler.scheduleTests (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@jest/core/build/TestScheduler.js:350:5)
at runJest (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@jest/core/build/runJest.js:376:19) {
  code: 'E_CREATING_EXECUTION_FAILED',
  originalError: Error: 
  at TestimExecutionsApi.createExecution (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@testim/root-cause-jest/node_modules/@testim/root-cause-core/lib/testim-services-api/testimExecutionsApi.js:45:50)
  at runMicrotasks (<anonymous>)
  at processTicksAndRejections (internal/process/task_queues.js:97:5)
  at RootCauseExecutionsApi.createExecution (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@testim/root-cause-jest/node_modules/@testim/root-cause-core/lib/testim-services-api/rootCauseExecutionsApi.js:49:9)
  at Object.persist (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@testim/root-cause-jest/node_modules/@testim/root-cause-core/lib/persist.js:42:34)
  at EnhancedDefault.onRunComplete (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@testim/root-cause-jest/lib/reporter/RunConclusion.js:58:13)
  at EnhancedDefault.onRunComplete (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@testim/root-cause-jest/lib/reporter/default.js:82:9)
  at ReporterDispatcher.onRunComplete (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@jest/core/build/ReporterDispatcher.js:88:9)
  at TestScheduler.scheduleTests (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@jest/core/build/TestScheduler.js:350:5)
  at runJest (/home/gilad/projects/obligo/frontend/test/e2e/node_modules/@jest/core/build/runJest.js:376:19)
}

Consider changesets based releases

mobx monorepo is using release workflow based on https://github.com/atlassian/changesets
Among other things, it has change log generation mobxjs/mobx#2616

Brainstorm: Can we automatically attach pages launched locally, with minimal code change?

Given the following example:
https://github.com/xtermjs/xterm.js/blob/master/test/api/MouseTracking.api.ts#L213-L219
The test runner is mocha
The browser and page are local variables in the module scope, and not global.
The before call is nested inside the describe
The page is created once, and re-used in all of the tests.

We need a way to wrap the page that is not based on global variables (patch playwright?), but also to associate it with the specific running test of mocha.

Document use with playwright-test

jest-playwright is now telling people to just use @playwright/test instead of Jest, eg:

import { test, expect } from '@playwright/test';

test.describe('headed block', () => {
  test.use({ headless: false });

  test('my headed test', async ({ page }) => {
    // ...
  });
});

$ npx playwright test

Currently there don't seem to be docs or examples on how to use root-cause with this test runner.

README: Link GIF screencast to a full video screencast

The README includes a small screencast as an animated GIF. Clicking on the GIF links to the GIF itself which is quite low resolution and hard to view. How about linking to full hosted video instead? Cheers!

networkLogs not captured (jest-playwright)

With or without networkLogs explicitly enabled in config, I don't see them captured. The networkLogs.har files have no entries.

I've set some breakpoints and I do see the listeners attached here:

root-cause/packages/root-cause-core/lib/networkLogs.ts

Line 113 in 0e8c037

cdpSession.on(eventToListenOn, listener);

But the listeners never get called, which leads me to believe the private page._delegate._mainFrameSession._client might have changed in recent Playwright versions.

I do see it used internally still at:

https://github.com/microsoft/playwright/blob/97be66b15c24246aff4ca2404de902952dbd82cb/src/server/chromium/crBrowser.ts#L215

Versions:

• "playwright": "1.7.1" (latest)
• "jest-playwright-preset": "^1.4.4" (latest)
• "@testim/root-cause-jest": "^0.1.30" (latest)
• Node v14.15.4

Custom screenshots directory

Hi,
How can I save the screenshots in another directory? e.g. I want all the runs under C:\RootCauseRuns and not under project\.root-cause\runs

Show localStorage/cookies in root-cause

Add a new hook to show localStorage/cookies

Integration example doesn't work: jest-and-puppeteer

Root Cause Integration Error

  at Object.ensurePrerequisite (node_modules/@testim/root-cause-jest/lib/helpers.js:20:11)

Error: ENOENT: no such file or directory, open 'C:\root-cause-main\integration-examples\jest-and-puppeteer.root-cause\runs\1613495584048\runConclusion.json'

Visual Studio Code Plugin

Hey,

On a call with @auchenberg (thanks again by the way!) he mentioned that when he was working on vscode a lot of times a goal was getting feedback closest to the code.

We should consider developing a VSCode plugin that adds the screenshots (and direct links) inside the test in VSCode, dumps the console in vscode's console and adds a link to cloud runs if the user has cloud runs active.

AGPL why?

Is there a reason behind the AGPL license choice?

• 8052f22

root-cause cli may be not available when not installing root-cause-core directly

When installing @testim/root-cause-jest, the root cause cli binary from core is not linked.
We probably want to the bin entry to jest & mocha packages

Support (page|frame|handle).evaluateHandle rect extraction/selector extraction

We do not collect rect for:
https://github.com/smooth-code/jest-puppeteer/blob/master/packages/expect-puppeteer/src/matchers/toMatchElement.js#L94-L102

Screenshot cache with previous test?

Hi there,

I am using the jest-puppeteer example shown in integration-examples/another-jest-and-puppeteer/example.test.js and I been noticing that the first screenshot of every tests are based on last screen of the previous tests that are being run.

Attached the code and the root cause run result.

example.test.zip

.root-cause.zip

Network logs is empty

networkLogs.har file:

{
  "log": {
    "version": "1.2",
    "creator": {
      "name": "Testim",
      "version": "1.0",
      "comment": "Created during a test executed by Testim."
    },
    "pages": [],
    "entries": []
  }
}

Show framework state (react/angular/vue) in root cause

Integrate with angular/vue/react and show framework state in root-cause

Dead link for jest integration

Hello, this link: https://help.testim.io/docs/root-cause-jest-integration-guide is dead, leading to 404.

Data driven tests in Jest only show the last test in the group

Need to better support Jest data driven tests (.each on tests).

Root Cause Cloud #test passed is confusing

The number of tests passed when running a single test in a single file shows 2/2 but it should likely show 1/2

This happens it counts the file (and it shouldn't) in the runs list.

Ability to filter tests that didn't use puppeteer from root-cause ls

Currently even tests that had no actions are shown in root-cause ls

Reported by @talkl

Run tests also on node 14 (and maybe 10). 15 soonish.

We want to run our tests also and node 14, and maybe 10.

To make it work on node 14 (and 15)
Our tests snapshots contains stacktrace frames that are node version specific.
We can remove these stacktraces, as we don't care about these frames, and the tests should pass as-is also on node 14/15.

Node 10 It's a different story. we do not collect stacktraces there as it's does not support async stack traces,
And the snapshots are substantially different

Multiple browsers and viewports

Hey,

In a call with @auchenberg (thanks again by the way 🙇‍♂️ hopefully we meet in a conference when it's possible) he raised several interesting issues with one of the biggest ones being the ability to run multiple browsers/viewports automatically and making it easier to do mobile-first with Emulation.setDeviceMetricsOverride.

One thing we can consider doing is instrumenting the page (which we already do) and opening several pages in different viewports and then adding UI to support viewing in different viewports for example:

• The user adds root-cause
• In their .root-causerc.json.json they have a viewports field containing multiple viewports/devices (probably ones that puppeteer supports)
• Root cause sees that array and opens multiple pages instead of one and calls Emulation.setDeviceMetricsOverride on them
• When an action is run (like page.click) it is sent to all pages.
• When a non action is performed (like extract text) it happens on one page?
  • Alternatively can we convert a jest/mocha test to a "data driven" test (like jest.each) in the background?

Issues:

• What's the UI like?
  • I imagine just a drop-down of all the devices that ran
• Running the test multiple times impacts state, this is challenging without mock network, would it be helpful to run it once, mock the network and then run it other times with mocked network?

adding CHANGELOG

Hey guys, awesome project! thanks!

How do you feel about using conventional commits and producing a changelog file? (so people will see what's changed between versions)

I don't mind issuing a PR for that (I believe it will be a one liner 😄 )

What do you think?

If Testim Editor extension is not installed - user gets prompted in Testim Cloud

Need to not require extension installation for the root-cause viewer. This broke sometime in the last few weeks.

chrome-har fork is `git+https` dependencies, some environments dose not have git

The chrome-har fork we are using is not published to NPM, but installed via git+https url from github.
Some node environments, such as some docker images on CI, dose not have git installed, and installing root-cause will fail.

Example:
https://app.circleci.com/pipelines/github/projectstorm/react-diagrams/76/workflows/f96be8f3-aedd-449b-94e1-3e140c5d545d/jobs/541

Support page.mainFrame().evaluateHandle('document')

Calls to page.mainFrame() and Element handles acquired by frames, are not instrumented

As in use in:
https://github.com/hoverinc/playwright-testing-library/blob/master/lib/index.ts#L147-L153

Using root-cause jest integration with typescript (ts-jest)

Hi, i'm trying to use Root Cause Jest integration and see the results locally, but because i'm using a page of puppeteer-cluster i suspect the results are not logged to root cause, when i tried to use the page, as shown in the jest example, it couldn't understand where page variable is defined (I'm using typescript).
So when i open npx root-cause show i see nothing.

Thank you guys

HTML/DOM Collection

Talking to @gioragutt regarding HTML collection and writing things down:

• add a per-step hook (like screenshots/console-logs) that calls Page.captureSnapshot as mhtml and save it (don't use fast-mhtml yet). Make sure to add it to the config file and make sure it's off by default.
• Open a PR.
• On top of it - add fast-mhtml and add a step (in the viewer) that parses the mhtml using it to display the output of Page.captureSnapshot.

Takeaways:

• Probably better to use Page.captureSnapshot and not DOMSnapshot given the standard format and the lack of info on parsing the output of DOMSnapshot.
• Probably better to save the MHTML as a separate file like screenshots rather than inline it in the results file.

Cannot get root-cause to work with jest-circus

I'd like to use jest.retryTimes() but that requires the circus runner.

The Root Cause docs suggest it needs jasmine2: https://help.testim.io/docs/getting-started-with-root-cause#test-runner-mode-setup

I did see #55 which seems to add support for circus, but when I follow https://github.com/testimio/root-cause/pull/55/files#diff-c0c48d2d9bb3d94e663e5058180363262a1b0193aa89d34c18cff0af38227ff2R71-R77 I get this warning on all runs:

(node:65990) UnhandledPromiseRejectionWarning: TypeError: this.summaryReporter.onTestCaseResult is not a function
    at EnhancedDefault.onTestCaseResult (/Users/fokkezb/Code/workflows/node_modules/@testim/root-cause-jest/lib/reporter/default.js:70:30)
    at ReporterDispatcher.onTestCaseResult (/Users/fokkezb/Code/workflows/node_modules/jest-cli/node_modules/@jest/core/build/ReporterDispatcher.js:78:24)
    at /Users/fokkezb/Code/workflows/node_modules/jest-cli/node_modules/@jest/core/build/TestScheduler.js:317:36
    at /Users/fokkezb/Code/workflows/node_modules/emittery/index.js:260:13
    at Array.map (<anonymous>)
    at Emittery.Typed.emit (/Users/fokkezb/Code/workflows/node_modules/emittery/index.js:258:23)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:65990) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 7)
(node:65990) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

And on failed runs it doesn't retry but errors with:

    To open in Root Cause viewer, run: npx root-cause show 2f9d8ed8e31fb646356e27d28f652f0a
     TypeError: this.summaryReporter.onTestCaseResult is not a function

      at EnhancedDefault.onTestCaseResult (../../../node_modules/@testim/root-cause-jest/lib/reporter/default.js:70:30)
      at ../../../node_modules/emittery/index.js:260:13
          at Array.map (<anonymous>)
      at Emittery.Typed.emit (../../../node_modules/emittery/index.js:258:23)
      at processTicksAndRejections (../internal/process/task_queues.js:93:5)'

When do you expect to provide full and official support for the circus runner?

Runtime error in extractCodeLocationDetailsSync when there's an issue with stack trace acquire

      at Object.extractCodeLocationDetailsSync (node_modules/@testim/root-cause-jest/node_modules/@testim/root-cause-core/lib/utils.js:311:47)
      at Object.sync (node_modules/@testim/root-cause-jest/lib/helpers.js:166:59)
      at Proxy.toMatch (node_modules/@testim/root-cause-jest/lib/hookExpect.js:83:39)
      at Object.<anonymous> (helpers/describe.js:113:19)

Get ready for npm 7

npm 7 is going to automatically install peer dependencies.
Currently we have both puppeteer and playwright as peer deps,
and we expect the user to install & use only one of them.
The coming npm 7 feature will also the other package for the user, which is not desirable.

Measure hooks timing + actual step timings, make some performances report out of it [Behind a flag]