creevey/creevey

Cross-browser screenshot testing tool for Storybook with fancy UI Runner.

IMPORTANT Looking for any help with maintaining

Cross-browser screenshot testing tool for Storybook with fancy UI Runner.

Core features

• 📚 Integrates with Storybook
• 📜 Uses stories as tests
• ✏️ Allows write interaction tests
• ✨ Has fancy UI Runner
• 🐳 Supports Docker
• ⚔️ Cross-browsers testing
• 🔥 Tests hot-reloading
• ⚙️ CI Ready

It named after Colin Creevey character from the Harry Potter universe.

Contents

• Pre-requisites
• How to Start
• Comparison with other tools
• Config/Options
• Examples
• Advanced usage
  • Write tests
  • Use your Selenium Grid (LambdaTest/BrowserStack/SauceLabs/etc)
• Future plans
• Known issues
• Used by

Pre-requisites

• Make sure you have installed Docker. But if you going to use your own separate Selenium Grid, you don't need Docker.
• Supported Storybook versions: ^6.4.0.

How to start

• Install creevey package

yarn add -D creevey

• Add addon creevey into your storybook config

// .storybook/main.js
module.exports = {
  stories: [
    /* ... */
  ],
  addons: [
    /* ... */
    'creevey',
  ],
};

• Start storybook and then start Creevey UI Runner in separate terminal. (To start tests from CLI, run Creevey without --ui flag)

yarn start-storybook -p 6006
yarn creevey --ui

And that's it.

NOTE: In first run you may noticed, that all your tests are failing, it because you don't have source screenshot images yet. If you think, that all images are acceptable, you may approve them all in one command yarn creevey --update.

NOTE: Creevey captures screenshot of the #root element and sometimes you need to capture a whole browser viewport. To achieve this you could define captureElement Creevey parameter for story or kind. Or you may pass any different css selector.

// stories/MyModal.stories.tsx

// NOTE: Define parameter for all stories
export default {
  title: 'MyModal',
  parameters: { creevey: { captureElement: null } },
};

// NOTE: Or define it for specific one

export const MyModalStory = () => <MyModal />;
MyModalStory.parameters = { creevey: { captureElement: null } };

Comparison with other tools

Features\Tools	Creevey	Loki	Storyshots	Hermione	BackstopJS	Percy/Happo	Chromatic
Easy-to-Setup	✔️	✔️	⚠️	⛔	⛔	✔️	✔️
Storybook Support	✔️	✔️	✔️	⛔	⛔	✔️	✔️
Run tests from Storybook UI	✔️	⛔	⛔	⛔	⛔	⛔	⛔
Cross-browser	✔️	⚠️	⛔	✔️	⛔	✔️	✔️
Test Interaction	✔️	⛔	⚠️	✔️	✔️	⛔	⛔
UI Test Runner	✔️	⛔	⛔	✔️	✔️	✔️	✔️
Built-in Docker	✔️	✔️	⛔	⛔	✔️	⚠️	⚠️
Tests hot-reload	✔️	⛔	⛔	⛔	⛔	⛔	⛔
OSS/SaaS	OSS	OSS	OSS	OSS	OSS	SaaS	SaaS

Future plans

• Allow use different webdrivers not only selenium, but also puppeteer or playwright.
• Add ability to ignore elements.
• Allow to define different viewport sizes for specific stories or capture story with different args.
• And more, check TODO for more details. Also feel free to ask about feature that you want

Known issues

Creevey is trying to build storybook but fail or tests don't work

This might happens because Creevey patches storybook webpack config and build nodejs bundle with stories meta information. And in some cases Creevey couldn't properly remove all unnecessary code cause of side-effects in stories files or you create stories dynamically. Try to rewrite such places. If it still doesn't help, send to me bundle that Creevey created (it located in node_modules/creevey/.cache/creevey/storybook/main.js)

A little bit later I'll add possibility to run tests without building that bundle, so it fixes this issue.

Docker-in-Docker

Currently it's not possible to run Creevey in this configuration. I'll fix this in later versions. If you use CircleCI or another CI that use docker to run jobs. Try to configure to use virtual machine executor

Update I added support to use local browsers. So it should be possible run Creevey inside docker. The only issue, that you need to find or build docker image with node, browser and selenium-webdriver. I'll add special images for Creevey later.

Chrome webdriver + 1px border with border-radius.

This cause to flaky screenshots. Possible solutions:

• Increase threshold ratio in Creevey config diffOptions: { threshold: 0.1 }
• Replace border to box-shadow border: 1px solid red -> box-shadow: 0 0 0 1px red
• Set max retries to more than 5

You can't directly import selenium-webdriver package in story file

Because tests defined in story parameters and selenium-webdriver depends on nodejs builtin packages. Storybook may fail to build browser bundle. To avoid import use these workarounds:

• .findElement(By.css('#root')) -> .findElement({ css: '#root' })
• .sendKeys(Keys.ENTER) -> .sendKeys(this.keys.ENTER)

Used by

License

creevey's Issues

Problems with skip options

Hi everyone!

I have some problems with skip option. Here is my config:

addDecorator(
  withCreevey({
    captureElement: TEST_CSS_SELECTOR,
    skip: { kinds: ["Editor"] }
  })
);

I expected to skip stories with "Editor" name, but instead it's locked all my stories.

Any help?

Tests started to fail randomly

Hi, thank you for this great lib you made!
Today almost all our tests started to fail locally and in GitHub Actions.
All the failures looks the same:

We are using v0.7.29.

Any ideas what can be the cause? I looked at selenoid and it seems haven't changed since March.

Error: ENOENT: no such file or directory, open 'C:\Users\User\myproject\report\tests.json'

Hi, thanks for this awesome project! I am trying to use it in my project and I am getting the following error:

It seems to me that before writing the file, you need to check if such a path exists. I fixed this by manually creating the "report" directory.

Creevey with Docker Desktop alternatives

Hi, since Docker Desktop changed its pricing, we need to use alternatives. I tried Rancher Desktop which seems to be good replacement however Creevey is not working with it, it just hangs with this output:

I also tried to setup Docker inside WSL and Creevey still hangs the same.

Both of these methods works fine with BackstopJS which is just using plain docker run. I expect that an issue can be in dockernode but I really lack knowledge in your code base and docker overall so I can't properly inspect.

1.0 goals

• Monorepo
  • chai-images
  • creevey
  • creevey-ui
• Browser adapters
  • creevey-selenium
  • creevey-puppeteer
• Storybook integration (Support all storybook frameworks)
  • creevey-react
  • creevey-vue
• UI
  • Rewrite UI
  • Improve performance for large amount of tests

Allow use creevey without storybook

Mobile test using selenoid chrome-mobile

Is it possible to use the chrome-mobile selenoid container?

Would be really nice if creevey could do the testing in mobile too

I gave it a few tries, but couldn't figure it out how to make it work

Compatible with Webpack 5?

Hello,

I'm attempting to replace Storyshots with Creevey. When I start up Storybook it fails to launch and I get this error:

/Users/ccreeger/sites/website/node_modules/webpack/lib/DefinePlugin.js:549
						const oldVersion = compilation.valueCacheVersions.get(name);

I have Storybook configured to use Webpack 5. Could that be the problem?

Thanks!

"Cannot use import statement outside a module", creevey doesn't see stories

In this repo, https://github.com/goright-io/hands-on-design-system

I am having the following error

There was no such error in other projects, and the storybook seems to be the same version. Could it be some version mismatch?

Cannot start test with a service like BrowserStack.

Hello.

I tried to do a multi-browser test using a service like BrowserStack.

However, due to the limited number of BrowserStack parallel connections, Creevey did not start.

Creevey connects in parallel with all registered Selenium nodes.

In a situation where the number of connections is limited, I think the following considerations are necessary.

1. Do not connect to Selenium Grid at UI startup. Connect when running the test.
2. Limit the number of connections to the Hub.
3. Sort the test execution order by browser(Because it costs a lot to start Node).

Filter

add filter for substring (browser, component, error)

Unexpected story export "image": expected function, received "object".

Recently we are seeing the following message when trying to run creevey together with storybook:

Unexpected story export "image": expected function, received "object".

These are parts of the output messages we are receiving

...
[Creevey:102325] INFO => Starting Master Process
Unexpected story export "image": expected function, received "object".
[Creevey:102514] INFO => Starting Worker for webkit.tablet
[Creevey:102520] INFO => Starting Worker for webkit.mobile
[Creevey:102507] INFO => Starting Worker for webkit.desktop
[Creevey:102527] INFO => Starting Worker for chrome.desktop
[Creevey:102542] INFO => Starting Worker for chrome.mobile
[Creevey:102548] INFO => Starting Worker for firefox.desktop
[Creevey:102535] INFO => Starting Worker for chrome.tablet
[Creevey:102555] INFO => Starting Worker for firefox.tablet
[Creevey:102557] INFO => Starting Worker for firefox.mobile
Unexpected story export "image": expected function, received "object".
Unexpected story export "image": expected function, received "object".
Unexpected story export "image": expected function, received "object".
Unexpected story export "image": expected function, received "object".
Unexpected story export "image": expected function, received "object".
Unexpected story export "image": expected function, received "object".
Unexpected story export "image": expected function, received "object".
Unexpected story export "image": expected function, received "object".
Unexpected story export "image": expected function, received "object".
[Creevey:102520] INFO => webkit.mobile:d3642382-81c8-4b80-9940-98254a7b3db4 is ready
[Creevey:102542] INFO => chrome.mobile:26f99aea78959ca0fff1d586d25b9139 is ready
[Creevey:102507] INFO => webkit.desktop:890f5a16-4356-4060-8930-79cfa71393e4 is ready
[Creevey:102527] INFO => chrome.desktop:1873166f41c9fe7c6b99dbdae03a0c5a is ready
[Creevey:102535] INFO => chrome.tablet:185ca2b3627406a5a8f40f098741be0a is ready
...

Is there anything we can do about it?

error if a filename starts with "creevey.config.js"

If we have a file named "creevey.config.js.js" the script fails.

I think the filename is checked only for starts and not for and exact match

Behaviour of waitForReady and delay

What is the behaviour of creevey if I set the following story parameters:

export default {
  parameters: {
    creevey: {
      waitForReady: true,
      delay: 2000,
    },
  },
}

Will creevey wait for the call of window.__CREEVEY_SET_READY_FOR_CAPTURE__.() and wait then 2000 ms to make the screenshot or will creevey make the screenshot immediately when window.__CREEVEY_SET_READY_FOR_CAPTURE__.() get's called?

Skipping mdx stories?

Hi,

Just wanted to ask if there's a way to skip mdx stories?

Error when starting storybook after installing Creevey

I have 2 clones of the same project. One has this error.

The difference is that on the working clone creevey is 0.7.14, on the broken one it is 0.7.17.

SyntaxError when starting creevey

After updating 0.0.30 -> 0.6.4 I've got the error at start.
storybook version - 5.3.17

[FAIL:28188] C:\billy\frontend-tasks\docs\node_modules@babel\runtime\helpers\esm\inheritsLoose.js:1
export default function _inheritsLoose(subClass, superClass) {
^^^^^^

SyntaxError: Unexpected token export
at Module._compile (internal/modules/cjs/loader.js:723:23)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module.load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at eval (webpack:///external%22@babel/runtime/helpers/esm/inheritsLoose%22?:1:18)
at Object.@babel/runtime/helpers/esm/inheritsLoose (C:\billy\frontend-tasks\docs\report\storybook\storybook.js:19860:1)
at webpack_require (C:\billy\frontend-tasks\docs\report\storybook\storybook.js:20:30)
[FAIL:28188] TypeError: Cannot set property 'innerHTML' of null
at showErrorDisplay (C:\billy\frontend-tasks\docs\node_modules@storybook\core\dist\client\preview\start.js:131:62)
at showException (C:\billy\frontend-tasks\docs\node_modules@storybook\core\dist\client\preview\start.js:162:3)
at StoryStore.renderUI (C:\billy\frontend-tasks\docs\node_modules@storybook\core\dist\client\preview\start.js:431:9)
at StoryStore.emit (C:\billy\frontend-tasks\docs\node_modules\eventemitter3\index.js:201:33)
at Timeout._onTimeout (C:\billy\frontend-tasks\docs\node_modules@storybook\client-api\dist\story_store.js:357:16)
at ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at listOnTimeout (timers.js:263:5)
at Timer.processTimers (timers.js:223:10)
error Command failed with exit code 4294967295.

Custom webpack loader not working

Hey!

If i use a custom webpack loader loader it works in storybook but not in creevy

import {stuff} from '!./myloader!./myfile-to-be-custom-loaded';

in creevy i get this console output

Cannot find module '!./myloader!./myfile-to-be-custom-loaded'

in storybook it works just fine.

Does creevy use the webpack config from storybook?
This case is actually even simpler since i explicitly invoke the loader so this does not even need the webpack config.
I also tried with absolute paths to avoid any issues with that. No success.

Any ideas?
Thank you!

Screenshot error in stories with popups/modals

I get an error for this story https://component-library-storybook.dc2-dev.elisa.fi/?path=/story/components-modal--size-s

The reason to it must be that div#root is 0 pixels of height. But that might be the case with all the modals/popups.

Does Creevey support the `storiesOf` syntax?

I've been trying to get this to work on our project (Storybook 6.3, Angular) and while I can run the Creevey UI, no stories are appearing.

Does Creevey work on storybook files which are defined using storiesOf, as opposed to the export default export const syntax?

We have developed our own DSL to simplify the definition of stories, and it uses storiesOf under the hood.

"" value for skip should mean true

This

creevey: {
    skip: ""
  }

equals to

creevey: {
    skip: false
  }

I think it has to be equal to true. Especially from the workflow perspective. I might want to have a reason for skipping but I don't YET have exact wording for it.

Waiting for Storybook readiness does not work in headless Firefox

Hello! I'm having difficulties running Creevey tests on Selenium with headless Firefox on Linux. Creevey is trying to start firefox and after a minite it fails with timeout

[CreeveyWorker]: Starting firefox:31362
[FAIL:31362] ScriptTimeoutError: Timed out after 60000 ms
    at Object.throwDecodedError (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/error.js:517:15)
    at parseHttpResponse (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/http.js:642:13)
    at Executor.execute (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/http.js:568:28)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at mixin.execute (/Users/nitive/Work/s7_ds/node_modules/selenium-webdriver/lib/webdriver.js:731:17)
    at runSequence (/Users/nitive/Work/s7_ds/node_modules/creevey/lib/server/utils.js:156:22)
    at getBrowser (/Users/nitive/Work/s7_ds/node_modules/creevey/lib/server/selenium/browser.js:505:5)
    at Object.worker (/Users/nitive/Work/s7_ds/node_modules/creevey/lib/server/worker/worker.js:196:19)

I started to debug what causes the timeout and found out that it is that peace of code

function waitForStorybook(browser) {
  return browser.executeAsyncScript(function (callback) {
    if (document.readyState == 'complete') return callback();
    window.addEventListener('load', function () {
      callback();
    });
  });
}

It turns out load event does not gets fired most of times for some reason and readyState stays in interactive state. I confirmed it by changing code like this

function waitForStorybook(browser) {
  return browser.executeAsyncScript(function (callback) {
    if (document.readyState == 'complete') return callback('readyState complete');

    window.addEventListener('load', function () {
      callback('load event');
    });

    setTimeout(function() {
    	callback('timeout. readyState: ' + document.readyState + ', loadEventEnd: ' + performance.timing.loadEventEnd)
    }, 50000)
  }).then(callbackReason => {
  	console.log(callbackReason) // Logs `timeout. readyState: interactive, loadEventEnd: 0`
  });
}

The issue happens only on headless Firefox on Linux. I tested the following machines

• Selenium Standalone on DigitalOcean droplet with Ubuntu 18
• Selenium Standalone on DigitalOcean managed Kubernetes with Debian 10
• Selenium Grid (Zalenium) on DigitalOcean managed Kubernetes with Debian 10
• Selenium Standalone on AWS managed Kubernetes with Amazon Linux 2

and reproduced bug on all of them.

I could not reproduced this bug with headless Chrome running on the same machines and Safari on other machine. Also sometimes load event does fire and everything works fine but it's quite rare.

Steps to reproduce

1. Create machine with Ubuntu on any cloud service (I have tested on DigitalOcean)
2. Run official firefox selenium-standalone docker image: docker run -it -p 4444:4444 --shm-size=2g selenium/standalone-firefox:3
3. Add browser configuration to creevey config file

browsers: {
  firefox: {
    gridUrl: `http://${vmIP}:4444/wd/hub`,
    browserName: 'firefox',
    'moz:firefoxOptions': {
      log: { level: 'trace' },
      args: ['--headless'],
    },
  },
},

4. Run creevey npx creevey --ui
5. Wait for 1 minute
6. Result: selenium returns timeout error (with status code 500 for some reason, but response body clearly states that it's a timeout error)

What to do

I'm not sure but it seems like we can use DOMContentLoaded event (and document.readyState === "interactive") instead of load event because we do not need subresources to be loaded (there is no fonts or images expect for favicon). It should also make creevey start a bit faster

"unrecognised content at end of stream" for Mac/Windows team

The screenshots made under Unix (Mac) don't work on Windows. Every single story gave this error

[START:chrome.desktop:4468] Utilities/Typography/Link
[FAIL:chrome.desktop:4468] Utilities/Typography/Link
   Error: unrecognised content at end of stream
    at module.exports.SyncReader.process (node_modules\pngjs\lib\sync-reader.js:43:11)
    at module.exports (node_modules\pngjs\lib\parser-sync.js:68:10)
    at Object.exports.read (node_modules\pngjs\lib\png-sync.js:7:10)
    at compareImages (node_modules\creevey\lib\server\worker\chai-image.js:48:39)
    at assertImage (node_modules\creevey\lib\server\worker\chai-image.js:99:11)
    at Proxy.matchImage (node_modules\creevey\lib\server\worker\chai-image.js:107:28)
    at Context.storyTest (node_modules\creevey\lib\server\stories.js:43:5)

Perhaps it is related to pngjs/pngjs#235 ?

We use

node v14.15.4
npm 6.14.4
creevey 0.7.28

Recommended way to disable animations in creevey

Currently I am fighting with some flaky tests in a larger project and I am wondering what @wKich would recommend as a way to disable web animations in the storybooks?

Is there something along the lines like reg-viz/storycap#8 for creevey?

Storybook 6.4.0 - TypeError: Cannot read properties of undefined (reading 'removeAllListeners')

When updating my project to storybook 6.4.0 I am getting a TypeError: Cannot read properties of undefined (reading 'removeAllListeners') error message when running creevey:

DevToolsActivePort file doesn't exist - Mac OS

Hi, i have a storybook project, installed creevey
started storybook
and, when I run creevey, (yarn creevey --ui)

It shows this:
[CreeveyMaster]: Pull docker images
✔ aerokube/selenoid:latest-release: Pull complete
[CreeveyMaster]: Pull docker images
✔ selenoid/chrome:latest: Pull complete
[CreeveyMaster]: Starting with pid 1029
[CreeveyWorker]: Starting chrome:1030
[FAIL:1030] WebDriverError: unknown error: Chrome failed to start: crashed.
(unknown error: DevToolsActivePort file doesn't exist)

Anyone knows how to solve this? I've looked all over the place, but found no working answer.

I'm running this over a Mac mini with Big Sur 11.4 and M1 Chip

Browser specific delays for tests

I am running into timing issues when doing the screenshots especially with webkit compared to firefox and chrome. The delay parameter somewhat brings relief to this, but when using the delay for a test it would slow down all the tests in all other browsers aswell.

We are testing mobile/tablet/desktop resolutions in chrome/firefox/webkit - so we have 9 different versions of the same test. The delay (e.g. 2000 ms) would be a total delay of 18.000 ms (9 x 2.000ms) for us, which adds up significantly for our testing pipeline.

Would it be possible to delay the test only for specific browsers and introducing a syntax comparable to the skip parameter. E.g.

 parameters: {
    creevey: {
      delay: {
        in: ["webkit.mobile", "webkit.tablet", "webkit.desktop"],
        ms: 2000
      }
    }
  }

UnknownCommandError: Unrecognized command: actions - Saucelabs setup

I am currently testing creevy with a bigger project and so far it works very well. Locally all tests run through smoothly, but when I am trying to connect to our saucelabs selenium grid over the saucelabs connect proxy I am receiving the following UnknownCommandError: Unrecognized command: actions error:

Does somebody have some hint for me or has somebody a working setup with saucelabs and creevey?

Fix Storybook output warning `Unexpected loaded state. Did you call `load` twice?`

I think it could be fixed if some of storybook modules would be deleted from require.cache and loaded again.

Readlink don't work on windows

Need to change storybook framework detection.

Print selenium grid agent hostname in test reporter output

[Logs] убрать лишнее

`ssh2` vulnerable version through `docker-modem`

Dependabot found that creevey has vulnerable ssh2 version that comes from docker-modem.
GHSA-652h-xwhf-q4h6

I see that docker-modem has fixed this issue in path release 3.0.3: apocas/docker-modem@0b24b61#diff-7ae45ad102eab3b6d7e7896acd08c427a9b25b346470d7bc6507b6481575d519

mocha-junit-reporter output only last testcase

I'm trying to use mocha-junit-reporter, but get file with only last testcase. Also i tried xunit reporter (with hack worker.js to add output to reporterOptions) and get same result.

Ability to pass a reporterOptions might also be useful.

Reference screenshots not browser that is not in configuration (cache)

My config is like the following

module.exports = {
  screenDir: path.join(__dirname, '../tests/visual/references/'),
  reportDir: path.join(__dirname, '../tests/visual/report'),
  diffOptions: { threshold: 0.1 },
  browsers: {
    "chrome.desktop": {
      browserName: 'chrome',
      // Define initial viewport size
      viewport: { width: 1366, height: 1000 },
    },
    "chrome.tablet": {
      browserName: 'chrome',
      // Define initial viewport size
      viewport: { width: 640, height: 1000 },
    },
    "chrome.mobile": {
      browserName: 'chrome',
      // Define initial viewport size
      viewport: { width: 480, height: 768 },
    },
  },
};

But when I do creevey --update, I get unnecessary references for not anymore existing "chrome" browser. Looks like some cache issue. It is possible to clean the cache automatically?

feat: ci mode, exiting after first fail

Hey there, great plugin!
I want to feature request a "CI" mode that make creevey exit on first error with a status code indicating failure. This way not all other images have to be checked and CI server time is saved.

maxRetries and non-existing screenshots

We are using creevey in our ci pipeline runs together with a maxRetries: 6 setting.

Whenever we are creating a new story and thus the reference screenshot is not existing yet, we are getting back a Expected image does not exists message, which is fine. Together with a maxRetries: 6 setting however, it is retried up to 6 times to find the reference screenshot. I think those screenshots that are not found should not be retried, despite a maxRetries setting in creevey.config.js.

Highlight current test

Подсвечивать открытый в данный момент тест

button check for tests with errors

[Creevey:21879] ERROR => TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined

Hi

Tried to run Creevey (on Linux) and have an exception described in the title.

I follow the instruction from the readme:

1. add the creevey
2. add it to the storybook config
3. run the storybook
4. and then run the creevey

Stack:

yarn creevey --ui

[Creevey:21879] INFO => Pull docker images
✔ aerokube/selenoid:latest-release: Pull complete
[Creevey:21879] INFO => Pull docker images
✔ selenoid/chrome:latest: Pull complete
[Creevey:21879] ERROR => TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at new NodeError (internal/errors.js:322:7)
    at validateString (internal/validators.js:124:11)
    at Object.join (path.js:1148:7)
    at createSelenoidConfig (./.yarn/cache/creevey-npm-0.7.39-fe9bc3e472-71ab9c68a4.zip/node_modules/creevey/lib/cjs/server/selenium/selenoid.js:38:43)
    at Object.startSelenoidContainer (./.yarn/cache/creevey-npm-0.7.39-fe9bc3e472-71ab9c68a4.zip/node_modules/creevey/lib/cjs/server/selenium/selenoid.js:168:69)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Object._default [as default] (./.yarn/cache/creevey-npm-0.7.39-fe9bc3e472-71ab9c68a4.zip/node_modules/creevey/lib/cjs/server/docker.js:121:18)
    at async _default (./.yarn/cache/creevey-npm-0.7.39-fe9bc3e472-71ab9c68a4.zip/node_modules/creevey/lib/cjs/server/index.js:48:5)

Looks like something wrong here with getCreeveyCache, or maybe anywhere else?

Storybook 6.3 + Vite support?

Storybook 6.3 introduced Vite support. Vite replaces webpack for us in our toolchain, greatly improves build speed, and works great with Storybook as of 6.3.

We've been running creevey for some time on local machines and in CI, it works well for us. When I try to run creevey with this new Vite setup I get this error:

[CreeveyWebpack]: Starting with pid 46200
[FAIL:46200] TypeError: builder.overridePresets is not iterable
    at getWebpackConfigForStorybook_6_2 (/Users/adi/Documents/Projects/Terminal2/node_modules/creevey/lib/server/webpack/index.js:180:94)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Object.compile [as default] (/Users/adi/Documents/Projects/Terminal2/node_modules/creevey/lib/server/webpack/index.js:247:34)

I could be wrong but from these errors I'm getting it looks like creevey depends on the webpack configuration in Storybook. Will or does creevey support the new Vite features for Storybook?

Storybook addons not working when running creevey

Hi,

We have a functional Storybook with some standard addons for adding toolbar buttons and theme switching funtionality and a custom webpack config in order to support sass for storybook in my main.js

const path = require('path');

module.exports = {
	"stories": [
		"../src/**/*.stories.mdx",
		"../src/**/*.stories.@(js|jsx|ts|tsx)"
	],
	"addons": [
		"@storybook/addon-links",
		"@storybook/addon-essentials",
		"storybook-addon-themes",
		{
			name: '@storybook/addon-docs',
			options: {
			  configureJSX: true,  //  Needed for the mdx documentation files
			  babelOptions: {},
			  sourceLoaderOptions: null
			},
		},		
		"creevey"
	],
	webpackFinal: async (config, { configType }) => {
		// `configType` has a value of 'DEVELOPMENT' or 'PRODUCTION'
		// You can change the configuration based on that.
		// 'PRODUCTION' is used when building the static version of storybook.

		// Make whatever fine-grained changes you need
		config.module.rules.push({
			test: /\.*scss$/,
			use: ['style-loader', 'css-loader', 'resolve-url-loader', 'sass-loader'],
			include: path.resolve(__dirname, '../'),
		});

		// Return the altered config
		return config;
	}
}

When I run Creevey and refresh the page, my classes for theming applied on the body element seem gone and therefore the entire storybook has no styling anymore. Creevey takes snapshots of these unstyled elements for comparing but we can't use these in production.
Here the example of a button with Creevey not running:

This is the result after starting Creevey and refreshing the page:

I also see this warning but i'm not sure if it is related:

It says unable to find main.js in a strange path construction and a few lines after that it adds the stories defined in main.js so it must be able to find it I think?

Do you have any idea what could be causing this?

can't selectStory in IE.

[email protected]

https://github.com/wKich/creevey/blob/master/src/server/selenium/browser.ts#L306

This line is not transpiled.
IE doesn't support object spread syntax, so it doesn't work.

Fail test if story name contains invalid characters

Now, if story name contains character, which is not allowed in filename (e.g. :), test fails after timeout. In this case it would be more much more convenient if test would fail immediately and with corresponding error

Storybook vite builder breaks creevey chrome after hot-reload

I'm trying to setup creevey in a project using storybook with the vite builder.

On the first run after launching storybook and creevey ui it seems to work fine most of the time, however if I make a change in a story or component that triggers a reload on storyboook, and then run the tests again, I get this error in creevey:

Error: Creevey can't switch story. This may happened if forget to add `creevey` addon to your storybook config, or storybook not loaded in browser due syntax error.
    at selectStory (node_modules/creevey/lib/cjs/server/selenium/browser.js:376:27)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at Context.switchStory (node_modules/creevey/lib/cjs/server/selenium/browser.js:547:3)

I have a container running chrome and another with firefox, the firefox one seems to not break on a reload, but I had that error pop up on it too a few times, I just couldn't figure out a pattern yet.

If I run storybook with the normal webpack builder everything works perfectly, which is what I'll be doing for the moment, but it would be great if there was an easy fix for the issue as the storybook is quite big, and it takes a bit to refresh on changes with webpack

Using webkit/safari results in ERROR => UnknownCommandError: Unknown command: /wd/hub/session

My creevey.config.js looks like the following:

const path = require('path')

module.exports = {
  storybookUrl: 'http://localhost:9001',
  storybookDir: path.join(__dirname, '../storybook'),
  diffOptions: { threshold: 0.1 },
  maxRetries: 6,
  browsers: {
    'webkit.desktop': {
      browserName: 'webkit',
      viewport: { width: 1366, height: 768 },
      dockerImage: 'browsers/safari:14.0',
      version: '14.0',
  },
}

Unfortunately this leads to the following error:

[Creevey:126750] INFO => Starting Worker for webkit.desktop
[Creevey:126750] ERROR => UnknownCommandError: Unknown command: /wd/hub/session

The reason for this is IMHO https://github.com/wKich/creevey/blob/65bcb07cf8675e1f5cf24456c518196a383a68a3/src/server/selenium/selenoid.ts#L38
as webkit is not part of the browserNames that are checked there.

A temporary workaround is to use opera as browserName, then it works.

config

config for ports (may be smth extra)

failFast:true disables maxRetries config option

I am using creevey to run in our ci pipelines and fighting with some flaky tests/components, this is why I used a maxRetries: 5 setting. To get some faster feedback I also used the new failFast: true config option. Unfortunately it seems that failFast: true kind of disables the maxRetries config option and immediately fails on the first failing screenshot.

When both options are set I think it would be a more expectable behaviour when screenshotting is retried until maxRetries is reached and then creevey fails.

Creevey does not run on Windows machine

I am trying to set up Creevey at my project, and on Mac it works nicely. But my colleague with Windows have errors like the following. Our guess is that it has something to do with system security restrictions (which unfortunately cannot be changed).

If this is the problem, the possible solution would be to run needed docker containers separately in admin mode (we did the same for another testing setup). In Creevey documentation I could not find information related to the issue.

Is there any suggestion how to fix?

Do not reset diff view mode after switch between tests

было бы круто, если бы вьюшка запоминала вид который у тебя стоит, например своп и врубала его, а не всегда сайд-сайд

How to handle waiting for async content?

I have a page where some content is loaded from the network. The generated screenshot shows white parts, until I repeatedly click the play button to generate the screenshot again.

What is the recommended approach?