meinaart / cypress-plugin-snapshots Goto Github PK
View Code? Open in Web Editor NEWPlugin for snapshot tests in Cypress.io
License: MIT License
Plugin for snapshot tests in Cypress.io
License: MIT License
Thanks for creating this great plugin, to me, it's much better than another visual test plugin.
But...that another plugin provides an must-have environment setting to prevent failures but I can't find a similar feature in yours:
A feature like:
Run Cypress with --env failOnSnapshotDiff=false in order to prevent test failures when an image diff does not pass.
Is that possible you also provide this preventing failures setting in this plugin? Please.
In the current implementation it seems that the usual screenshot
options are stripped from the toMatchImageSnapshot
command.
cy.get(`.header-bar`).toMatchImageSnapshot({
clip: { x: 0, y: 0, width: 300, height: 50 }
});
This provides the following output:
CommandOptions
{
"autoCleanUp": false,
"autopassNewSnapshots": true,
"diffLines": 3,
"excludeFields": [],
"ignoreExtraArrayItems": false,
"ignoreExtraFields": false,
"normalizeJson": true,
"prettier": true,
"imageConfig": {
"createDiffImage": false,
"resizeDevicePixelRatio": true,
"threshold": 0.01,
"thresholdType": "percent"
},
"prettierConfig": {
"html": { "parser": "html", "tabWidth": 2, "endOfLine": "lf" }
},
"serverEnabled": false,
"serverHost": "localhost",
"serverPort": 2121,
"token": "2420dd834febe1b6ea977fa70d1303366cb535e53aa48cab6a509c8bf6029950543dd8ce47c8311a39601d65144510088f56e9f34fb658b7c87e24b7b47e9892",
"updateSnapshots": true,
"clip": { "x": 0, "y": 0, "width": 300, "height": 50 }
}
ScreenshotConfig
{"blackout":[".snapshot-diff"],"capture":"fullPage","log":false}
Is this expected behavior?
With the above combination snapshots are not being updated when the setting is set to true, removing the image running the spec works fine.
excludeFields
cannot be overridden in toMatchSnapshot
because it gets merged with settings from config
Hey, it's me again.
The current snapshots are saved in the relative path of the test spec file, under __snapshots__
folder. To improve the performance, we introduce an index_spec.js
to collect all test files in the same folder and run them together, now we have 2 issues:
index_spec.js
, all snapshots are saved in one index_spec.js.snap
, which is not easy to track when we have many cases.a_spec.js.snap
in diff path, which will be AUTOPASS because it's not existed before.I have an idea: generate the file name according to the key word in the test title, but it seems annoying since we need to unify all test titles and index_spec.js
is not in common use. So I would like to hear from you, you should have better vision.
Here is my proposal code change(the test title should follow the pattern - [Filename] XXXXXX):
function getSnapshotFilename(testFile, snapshotTitle) { const dir = path.join(path.dirname(testFile), DIR_SNAPSHOTS); const filename = `${snapshotTitle.match(/\[(.+)?\]/)[1]}.snap`; return path.join(dir, filename); }
Here is the context of index_spec.js
:
const reqCont = require.context('./', true, /_spec\.js?$/); reqCont.keys().forEach(filename => reqCont(filename));
Describe the bug
The plugin works fine when tests are launched from npx cypress open but it throws an error when they are launched using cypress run.
The error is:
<![CDATA[TypeError: Cannot read property 'height' of undefined at getImageData (https://chs2020-dev.ccs-chs-swf-qa.cegedim.cloud/__cypress/tests?p=cypress\support\index.js-194:15717:19) at Context.<anonymous> (https://chs2020-dev.ccs-chs-swf-qa.cegedim.cloud/__cypress/tests?p=cypress\support\index.js-194:15104:22)]]>
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The behavior should be the same as cypress open
Desktop (please complete the following information):
Note: it seems that the error happens on this function getImageData afterScreenshot.
Investigate blackout parts of screenshot
Describe the bug
warning cypress-plugin-snapshots > [email protected]: New code is available at github.com/uNetworking/uWebSockets.js
To Reproduce
yarn add --dev cypress-plugin-snapshots
Expected behavior
No warning should occur when installing.
Desktop (please complete the following information):
imageConfig
from cypress.json is not picked up correctly
Repro case here: https://github.com/jjt/cypress-plugin-snapshots-test
It seems like it only ever takes an initial snapshot and doesn't recapture it. For example, here's after the second run after some changes, where you can see that the contents of the snapshot is a red square with the words HELLO THERE, and the contents of the DOM on the second run is a green square with the text oh hi
Fix auto clean up (also when .only is used)
I'm seeing the following error when running cypress open
. I have configured plugins as specified.
Error running plugin
The following error was thrown by a plugin. We've stopped running your tests because a plugin crashed.
Error: listen EACCES 127.0.0.1:2121
at Object._errnoException (util.js:1024:11)
at _exceptionWithHostPort (util.js:1046:20)
at Server.setupListenHandle [as _listen2] (net.js:1334:19)
at listenInCluster (net.js:1392:12)
at GetAddrInfoReqWrap.doListen [as callback] (net.js:1501:7)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:97:10)
Hi Meinaart,
First, great thanks to your plugin. Our team is considering writing a similar plugin, and you just bring it out to open source, save us a lot of time.
But I encounter this issue during the cypress init step, it failed at the very beginning:
My guess is Cypress is running on browser, and which doesn't support fs
.
Project Background:
react 7, node v10.0
When matching a large snapshot, it can take long (10-30 seconds on regular amount of data in our application, which can mean 20-50k line snapshots, if that's any indication) to perform the task. Sometimes with a bit more data it even times out with our 2 minute task timeout. When this happens, the console has a lot of GET socket.io 500 errors, which seem to time out at 30 seconds each.
To me it would seem like there's a 30 second timeout for that socket.io get and if matching the snapshot regularly goes over that, it will never finish, but that's just my speculation without looking at the code.
Make toImageSnapshot
error configurable so it can include a link to image.
For example for junit output:
[[ATTACHMENT|/absolute/path/to/some/file]]
edge case:
In our Cypress example recipes repo https://github.com/cypress-io/cypress-example-recipes each separate application is in a subfolder and all dependencies are installed at the root level. Thus we only need to clone the repo, run npm install
from the root and then inside each examples/...
folder we can start the app and run Cypress tests. Example package json from branch that uses this plugin is testing-redux-store
and pull request cypress-io/cypress-example-recipes#211
The application is in examples/blogs__testing-redux-store
and its package.json
is just
{
"name": "blogs__testing-redux-store",
"version": "1.0.0",
"description": "Testing Redux store using Cypress",
"private": true,
"scripts": {
"start": "../../node_modules/.bin/react-scripts start",
"cypress:run": "../../node_modules/.bin/cypress run",
"cypress:open": "../../node_modules/.bin/cypress open"
},
"browserslist": [
">0.2%",
"not dead",
"not ie <= 11",
"not op_mini all"
]
}
(or at least it should be this)
"cypress-plugin-snapshots": "1.0.6"
is installed in the root folder.
But if I run tests I get an exception
Error: CypressError: Timed out retrying: cy.readFile("node_modules/diff2html/dist/diff2html.css") failed because the file does not exist at the following path:
/Users/gleb/git/cypress-example-recipes/examples/blogs__testing-redux-store/node_modules/diff2html/dist/diff2html.css
Because this error occurred during a 'before all' hook we are skipping all of the remaining tests.
I have not looked at the dependency loading code, but I know it can be in an unexpected place, see how I load CSS for cypress-dark
here https://github.com/bahmutov/cypress-dark/blob/0460aa71a23080780605936257896ed3f0777cb9/src/utils.js#L7 which is ugly.
I was thinking a better way would be to somehow to ask the plugin where it is installed - because it has access to the true __dirname
without browserify wrap. Then the front end code can correctly do its part, or even ask the plugin via cy.task
for the path to load.
Save screenshots per platform (optionally, because of text rendering)
Describe the bug
When clicking at "COMPARE SNAPSHOT", it opens a diff modal with Expected vs Actual.
But this only seems to work for the first snapshot. If there are other snapshots it doesn't open any modal (nothing happens)
To Reproduce
Steps to reproduce the behavior:
Have more than one image snapshot that fails and try to open diff modal
Expected behavior
A modal should be opened for each snapshot, showing its expected vs actual.
Desktop (please complete the following information):
the ignoreExtraFields functionality that is ignoring fields that are not in baseline snapshot file (for data) is not ignoring extra fields.
This was working in version 1.2.5 and is not working anymore in version 1.2.6
Describe the bug
If you use toMatchSnapshot();
after a cy.get()
that returned multiple elements, the final snapshot will look something like this: [{}, {}, {}]
To Reproduce
Steps to reproduce the behavior:
cy.get('div').toMatchSnapshot();
when there are mutiple div
s in the current context.Expected behavior
An array of stringified dom elements.
Desktop (please complete the following information):
We are trying to use your plugin and when we use the ImageSnapshot cypress throws an error.
Sample code:
cy.document().toMatchImageSnapshot()
The .png image snapshots are created correctly under __image_snapshots__ but the tests fail and cypress throws the following error:
CypressError: cy.task('cypress-plugin-snapshot:matchImage') failed with the following error:
> TypeError: Path must be a string. Received undefined
at assertPath (path.js:28:11)
at Object.relative (path.js:1267:5)
at getImageData (/Users/ioanagalan/Templafy/Templafy.AutomatedQualityAssurance/CypressTests/node_modules/cypress-plugin-snapshots/src/utils/image/getImageData.js:8:24)
at matchImageSnapshot (/Users/ioanagalan/Templafy/Templafy.AutomatedQualityAssurance/CypressTests/node_modules/cypress-plugin-snapshots/src/tasks/matchImageSnapshot.js:70:15)
at invoke (/Users/ioanagalan/Library/Caches/Cypress/3.3.1/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/child/task.js:37:14)
at /Users/ioanagalan/Library/Caches/Cypress/3.3.1/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/util.js:48:16
at tryCatcher (/Users/ioanagalan/Library/Caches/Cypress/3.3.1/Cypress.app/Contents/Resources/app/packages/server/node_modules/bluebird/js/release/util.js:16:23)
at Function.Promise.attempt.Promise.try (/Users/ioanagalan/Library/Caches/Cypress/3.3.1/Cypress.app/Contents/Resources/app/packages/server/node_modules/bluebird/js/release/method.js:39:29)
at Object.wrapChildPromise (/Users/ioanagalan/Library/Caches/Cypress/3.3.1/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/util.js:47:28)
at Object.wrap (/Users/ioanagalan/Library/Caches/Cypress/3.3.1/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/child/task.js:43:8)
at execute (/Users/ioanagalan/Library/Caches/Cypress/3.3.1/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/child/run_plugins.js:86:12)
at EventEmitter.ipc.on (/Users/ioanagalan/Library/Caches/Cypress/3.3.1/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/child/run_plugins.js:136:5)
at emitThree (events.js:136:13)
at EventEmitter.emit (events.js:217:7)
at process.<anonymous> (/Users/ioanagalan/Library/Caches/Cypress/3.3.1/Cypress.app/Contents/Resources/app/packages/server/lib/plugins/util.js:25:29)
at emitTwo (events.js:126:13)
at process.emit (events.js:214:7)
at emit (internal/child_process.js:772:12)
at _combinedTickCallback (internal/process/next_tick.js:141:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
Cypress version: 3.3.1
cypress-plugin-snapshots version: 1.2.6
@cypress/snapshot version 2.1.3
Worth mentioning is that we are running on mac.
Note: This only happens when screenshots do not exist and they are created for the first time.
Related:
Add overwrite
property to toMatchSnapshot
to overwrite current snapshot. Makes it easier then doing it by hand.
Make it possible to disable auto clean. Auto clean doesn't work nicely with a workflow where people sometimes use .skip
or .only
.
I'm using
I've seen that using of toMatchSnapshot
on html element will not match the snapshot if the snapshot file contains elements with backslash.
In my example input element had pattern with backslash:
<span class="input input--number input--required"> <input type="number" name="new_address.house_number" value="" pattern="\d*" required="" data-date-format="mm/dd/yyyy"/> </span>
Although in first iteration snapshot was created correctly and in every other test run before closing cypress runner, it passed successfully I guess due to caching of the snapshot file but after closing the runner and running the test again it failed due to mismatch while comparing with snapshot
backslash was not included in the snapshot:
- pattern="d*"
+ pattern="\d*"
I fixed this by adding additional backslash to the snapshot file manually because it would seem it's escaping it while doing the compare but I guess all backslash should be ignored while doing such comparison?
Snapshot files gets cached sometimes, so it compares with an older version then on disk.
A quickfix for now is to restart your Cypress when you made manual changes to your snapshot files.
The release 1.2.3 appear to be forcing the default browserify preprocessor. b36fb9c#diff-7aa2d6e03c8f7e1334d6724f903c6819R26
This breaks our project using @cypress/preprocessor-webpack
because no processing is done; so the JS syntax isn't valid and throw a parsing exception.
Typescript bindings need to be checked and updated where necessary.
Describe the bug
Error: listen EADDRINUSE: address already in use 127.0.0.1:2121 after installing plugin according to the README and then restarting cypress
To Reproduce
Steps to reproduce the behavior:
Clean up the
cypress.json
config so that it is valid json...
Error: listen EADDRINUSE: address already in use 127.0.0.1:2121
in the appExpected behavior
I would expect that the plugin works
Desktop (please complete the following information):
Hello! I've recently started to use both Cypress
and cypress-plugin-snapshots
and i have a doubt regarding fixed
elements, like for example a fixed Header
at the top of the page. In the docs says:
When passing fullPage to the capture option, Cypress scrolls the application under test from top to bottom, takes screenshots at each point and stitches them together. Due to this, elements that are position: fixed or position: sticky will appear multiple times in the final screenshot
for some reason, in different pages (which share some fixed components - in my case a fixed menu on the left) i'm getting different results, like this:
The first screen is correct: during scroll, the elements of the menu aren't repeated. On the second screen, after cypress scroll to the bottom, he's repeating some elements of the left fixed menu.
Screen were taken using cypress open
The weird part is that, sometimes, result changes if for example during test running, chrome is opened in fullscreen or not. Could be that cypress
have difficults, during screenshot elaboration, to mix repeated elements?
Hi guys, thanks for the great work of cypress-plugin-snapshots. I am currently working on implementing cypress-plugin-snapshots for my project, and I really want to customize __image_snapshots__
folder path.
Actually we use cypress during CI pipeline: as soon as the alpha(pre-release) deployment done, we will run cypress(and cypress-plugin-snapshots of course). In order to provide the snapshots' result, we have to configure __image_snapshots__
folder to a persist path in docker.
I consult the code source and supposed that making __image_snapshots__
folder path configurable is not hard and reasonable(at least by my side). Cypress also provide the ability to configure screenshotsFolder
and videosFolder
.
I am looking forward to your reply and ready to make a PR if it is needed.
Thanks again.
@meinaart could you publish a new version with the fix I provided last week?
Hi,
First: β€οΈ and π
Second: β€οΈ and π
Third: when checking config options like
function cleanUpSnapshots() {
const config = getConfig();
if (!config.autoCleanUp) {
return;
}
...
you cannot assume config
is not null (or maybe have logic in getConfig()
to merge user config with defaults).
right now for this cypress.json
file I get a crash
{
"baseUrl": "http://localhost:3000",
"ignoreTestFiles": [
"**/*.snap",
"**/__snapshot__/*"
]
}
:)
Based on patch functionality of diff-ing. Quite complicated to implement.
Displaying successful image snapshots does not always work, only the last one seems to work.
Describe the bug
I was developing a snapshots for all of my active pages (for visual regression testing automation) using UI $ cypress open
and viewport set to 1440x900
with cy.viewport('macbook-15');
on a global level.
When tried to run everything headless using cypress run
, started getting errors everywhere. Diff shown me, that actual
image snapshot differs from expected
via viewport size. After checking actual resolution of generated images, got following results:
expected
image size was as configured - 1440px x 900px
actual
image size was 1280βΓβ720
Tried already defining a viewport width/height explicitly inside cypress.json
or even passing as CLI parameters. None of them helped.
To Reproduce
Steps to reproduce the behavior:
beforeEach()
block inside existing test which uses cy.document().toMatchImageSnapshot();
cy.viewport(1440, 900);
inside of this beforeEach()
blockcypress open
(UI version of Cypress)1440x900px
Electron
with cypress run
diff
and actual
actual
image dimensions and you'll see that it's 1280βΓβ720
for some reason, ignoring viewport configuration.Same steps above can be repeated, while moving beforeEach() { cy.viewport(1440, 900); }
into global commands.js
, or even specifying them inside cypress.json
or passing as CLI
parameters into cypress run
.
Expected behavior
Images taken both during cypress open
or cypress run
are respecting custom viewport configuration.
In short there should be no difference in dimensions between images taken under cypress open
and cypress run
.
Desktop (please complete the following information):
Use patch to update snapshot not whole contents of snapshot, this would probably fix variables.
Describe the bug
Using .toMatchImageSnapshot() on a table, I'm getting a black square as the snapshot. Best shown by the following images:
This is the HTML for the table (using NextJS):
<table class="jsx-1824392073 table table-sm table-striped table-hover">
<caption class="figure-caption w-100 text-center">
<span style="font-style:italic">Table </span
>2<span style="font-style:italic">
Showing a data tabel. This is a tabel where the content is only text
and/or equations</span
>
</caption>
<tbody>
<tr>
<td class="jsx-795880060 "></td>
<td class="jsx-1311057988 ">
<span style="font-weight:bold">Year 1</span>
</td>
<td class="jsx-1311057988 ">
<span style="font-weight:bold">Year 2</span>
</td>
</tr>
<tr>
<td class="jsx-795880060 ">Category 1</td>
<td class="jsx-1311057988 ">3</td>
<td class="jsx-1311057988 ">5</td>
</tr>
<tr>
<td class="jsx-795880060 ">Category 2</td>
<td class="jsx-1311057988 ">4</td>
<td class="jsx-1311057988 ">4</td>
</tr>
<tr>
<td class="jsx-795880060 ">Category 3</td>
<td class="jsx-1311057988 ">5</td>
<td class="jsx-1311057988 ">6</td>
</tr>
</tbody>
</table>
I haven't got time at this very moment to reproduce a bare bones example, but will provide more info if this is not a known and common issue? (If you need more - I will look at the point below)
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Additional context
Add any other context about the problem here.
I have a snapshot with the value [ true, false ]
.
When I run:
describe('My First Test', () => {
it('Does nothing', () => {
cy.wrap(true).toMatchSnapshot();
});
});
I get an error message due to mismatching snapshot as expected.
When running:
describe('My First Test', () => {
it('Does nothing', () => {
cy.wrap(false).toMatchSnapshot();
});
});
This generates a pass, not going into the toMatchSnapshot() method. The last log displayed is "WRAP false".
Edit: this also happens to null
and undefined
.
Having the assertion separate from the test makes it less likely to be reviewed and makes it harder to align assertion value and test.
Describe the solution you'd like
To implement toMatchInlineSnapshot like jest.
Jest implementation: jestjs/jest@d3a6a74#diff-5d8368ae467702f37535e03f31a2ed96
It seems doable from a cursory look. I would follow the jest approach, requiring prettier if itβs used. Iβm interested in implementing this myself, just raising this in case you have objections or suggestions.
Is your feature request related to a problem? Please describe.
I have an app that is built with styled-components
so all classNames are generated randomly.
Describe the solution you'd like
Some option like ignoreAttributes
would fit.
That'd be nice to skip classNames so snapshots will work properly.
E.g. instead of this:
- <div class="sc-bdVaJa sc-bwzfXH Primitives__Page-sc-12hrqp0-4 gQrDDx">
18 - <h1
19 - font-size="calc(1.8em + 5vw)"
20 - font-weight="bold"
21 - class="sc-bdVaJa sc-htpNat sc-bxivhb jcKHLF"
22 - >
23 - Komfy
24 - </h1>
I would see this:
- <div>
18 - <h1
19 - font-size="calc(1.8em + 5vw)"
20 - font-weight="bold"
21 - >
22 - Komfy
23 - </h1>
Describe alternatives you've considered
Image snapshots
I have been following the README step by step. Here is the state of my dummy project: Repo
I wrote this test: File
cy.visit("localhost:3000")
.document()
.toMatchImageSnapshot();
Running that test causes a screenshot to be saved. However, when I change what is rendered on the page, the tests still pass!
I do not understand why the tests pass, if what is rendered in cypress' browser looks different than the screenshot that is saved.
I do not understand how toMatchImageSnapshot works. What is the base image that it uses? What is the current image?
Could someone please advise me how to get this test failing as expected?
Hi, thanks for the amazing plugin!!!
There is a possible bug, as when I run all tests in the cypress browser I receive following error:
CypressError: cy.task('cypress-plugin-snapshot:matchText') failed with the following error:
> Error: Cannot load snapshot file. File "__snapshots__/__all.snap does not contain valid JSON or javascript
Hi friends,
I am love the promise of this plugin. I am using typescript for my tests.
Currently when i run .toMatchSnapshot() it puts the snapshot in the dist folder.
Is it possible for us to configure it to point to a different target i.e. ../src
Is there a config setting somewhere that I can take advantage of?
===== second (and happy to make a separate issue) ===
be cool if we could add typings, or instructions on how to extend the cypress name space so I do not need to do
// @ts-ignore
cy.getByTestid('chart-engagement').toMatchSnapshot();
Thanks!!!!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.