visual-regression-tracker / visual-regression-tracker Goto Github PK

View Code? Open in Web Editor NEW

570.0 17.0 54.0 12.77 MB

Backend and Frontend application for tracking differences via image comparison

Home Page: https://visual-regression-tracker.com

License: Apache License 2.0

Shell 84.94% Dockerfile 15.06%

visual-testing visual-tests regression-tests image-comparison cypress playwright docker testing hacktoberfest

visual-regression-tracker's Introduction

Visual Regression Tracker

Open source, self hosted solution for visual testing and managing results of visual testing.

Hello

Take 1-minute survey to help us priorities tasks better ❤️
Twitter
Telegram group

How it works

Service receives images, performs pixel by pixel comparison with it’s previously accepted baseline and provides immediate results in order to catch unexpected changes.

Features

Automation framework independent - no need to stick with specific automation tool, integrate with existing one
Platform independent - web, mobile, desktop etc. as long as you could make a screenshot
Baseline history - track how baseline image changed since the beginning
Ignore regions - improve stability by ignoring not important or not controllable parts of image
Language support - JS, Java, Python, .Net or any other via REST API (need more?)
Easy setup - everything is inside Docker images
Self-hosted - keep your data save inside your intranet
Can compare PDF too - use standalone jar to compare pdf and images from a folder

Glossary

TestVariation - historical record of Baselines by Name + Branch + OS + Browser + Viewport + Device,
Baseline - validated and accepted image, latest will be used as expected result in TestRun
TestRun - result of comparing image against Baseline
Build - list of TestRuns
Project - list of Builds and TestVariations

Set up

Use any of the below ways to setup the server. Docker has to be installed on the machine.

Quick Setup

Linux, macOS, WSL

Download the installation script

curl https://raw.githubusercontent.com/Visual-Regression-Tracker/Visual-Regression-Tracker/master/vrt-install.sh -o vrt-install.sh
chmod a+x vrt-install.sh

Run the installation script

./vrt-install.sh

Command line arguments

Installs the Visual Regression Tracker

Usage: ./vrt-install.sh

Arguments:
    -h | --help
    -a | --frontend-url <url>   Set the Front-end url. Default: http://localhost:8080
    -r | --backend-url <url>    Set the API url. Default: http://localhost:4200
    --jwt-secret <secret>       Set the JWT secret. Default: randomly generated

Manual Setup

Copy docker-compose.yml

$ curl https://raw.githubusercontent.com/Visual-Regression-Tracker/Visual-Regression-Tracker/master/docker-compose.yml -o docker-compose.yml

Copy .env

$ curl https://raw.githubusercontent.com/Visual-Regression-Tracker/Visual-Regression-Tracker/master/.env -o .env

In .env file, ensure that the REACT_APP_API_URL has the right address. If it will be accessed from other machines, change localhost with IP or other resolvable name. Ensure the ports being used are free to use.
Start service

$ docker-compose up

Wait until you see your creds printed.

New users and projects could be created via frontend app by default on http://localhost:8080

Run VRT with logging enabled in Elasticsearch

This is for the users who want to monitor VRT logs via Kibana. It is expected to have basic knowledge of Elastic stack (especially Kibana) for the admin so that the logs can be managed and dashboards created in Kibana. Since logging will be retained by Elasticsearch, it will consume a little more memory and CPU. If you see error in the console, please consult Elasticsearch documentation.

It is recommended to run the program as root user which will ensure permission and ownership related issues will not have to be dealt with.

Clone or download this repository.
Move to the downloaded/cloned repository. In .env file, ensure that the REACT_APP_API_URL has the right address. If it will be accessed from other machines, change localhost with IP or other resolvable name. Ensure the ports being used are free to use.
Follow either of below sub steps.
- If your organization does not have Elasticsearch server running or if you want to start Elastic stack on your own, start service by giving below command.
  
  $ docker-compose -f docker-compose.yml -f docker-compose.elastic.logging.yml up
- If you want to re-use the already running Elasticsearch server in your organization, go to filebeat/config/filebeat.yml and edit hosts to point to the Elasticsearch server. Also, point ELASTIC_URL to this server in .env file. Start service by giving below command.
  
  $ docker-compose -f docker-compose.yml -f docker-compose.logging.yml up
If you are not using root user, in some OS, you may see an error Exiting: error loading config file: config file ("filebeat.yml") must be owned by the user identifier (uid=0) or root. In that case, press Ctrl+C, and follow Elasticsearch instructions. Once done, start the service again.

Integration

Use implemented libraries to integrate with existing automated suites by adding assertions based on image comparison. We provide native integration with automation libraries, core SDK and Rest API interfaces that allow the system to be used with any existing programming language.

Agents

Core SDK

Basic wrapper over API to be used for integration with existing tools

Getting started guide

Videos

How to set up on Remote machine via ssh

Wiki

Integration examples

Here you could find examples

Contribution

Try it, raise tickets with ideas, questions, bugs and share feedback :)
More language support for SDK
More integration with specific testing frameworks (agents)

Contributors ✨

Thanks goes to these wonderful people (emoji key):

_{Pavel Strunkin} 💻 💼 🤔 🔌	_{Daniel Crowe} 🔌 👀	_{Surat Das} 💻 🔌	_{Oleksandr Romanov} 🔌	_{Terentev Denis} 🔌	_{JustSittinHere} 🔌	_{Dekara VanHoc} 🔌
_maddocnc 💻	_{Aaron Chelvan} 💻 📖	_marcm-qa 🔌	_Eduard-iCH 🔌	_Roman 💻	_{Dimitri Harding} 💻	_vkostromin94 🔌
_{Bruno Ferreira} 💻	_{Loïc PÉRON} 💻 🔌	_{Alexey Volkov} 🔌 💻	_{Egor Lipskiy} 🔌	_nitschSB 💻	_polyvisual 💻	_{Juga Paazmaya} 🔌 💻

This project follows the all-contributors specification. Contributions of any kind welcome!

visual-regression-tracker's People

Contributors

Stargazers

Watchers

Forkers

nick130589 syesychko nigam-ashish sapugatt paychokxxx jradom dcrowe schanne-dhbw flashcsgroups cbruyndoncx danielosg dortizesquivel mikuu indhumathi-unifi twerema svasenkov maddocnc nathanclevenger mrgklwong gcaaa31928 sumanthmattapali mohamedraslan kevorld tavisca-chaitalishah giovyline marcel-veselka imademethink lashchanka andresz74 mohananup elaa0505 jeff526 brendanmckeown zhl463 tchiaspko tobmaster wooodhead snejlalelu mohammedshakeer555 prasanta-biswas maayehkhaled bbromaniovlev clgwlg vladimirvsg vuchkov mubbashir tanwangw polyvisual chrisldone kappa0x animeshparida luckhiem chenhaijun02 xxdm

visual-regression-tracker's Issues

Frontend. Link from context menu always leads to last screenshot

Case:

Write 3+ tests on difference pages and launch it
Go to web and try open any of table row with context menu "Details" (see screenshot)
All of tests details will contains screenshot of last made test

But its ok if click on table row itself

TestRun. Do not change baseline image on approve, add status for approved tests

to display previous baseline state vs accepted

Build list. Add loading animation

TestRun. Add link to TestVariation

TestRun list. Add loading animation

Move release process to Github Actions

TestVarition history. Device is not shown

Do not create diff image in case no diff found

TestRun details. Draw new ignore area instead of adding fixed rectangle

Add examples for integration over API

TestRun. Change mouse cursor on hover/drag image inside canvas

TestRun. Add option to comment

Should be used as additional information that helps to manage test results

Exception if image size do not match to baseline

Steps:

Create test with baseline accepted
Upload image with another dimension

Exception:

api_1        | [Nest] 17   - 05/21/2020, 9:13:40 PM   [ExceptionsHandler] Image sizes do not match. +226762ms
api_1        | Error: Image sizes do not match.
api_1        |     at Object.pixelmatch [as default] (/node_modules/pixelmatch/index.js:21:15)
api_1        |     at TestRunsService.create (/dist/test-runs/test-runs.service.js:117:60)
api_1        |     at TestService.postTestRun (/dist/test/test.service.js:29:52)
api_1        |     at processTicksAndRejections (internal/process/task_queues.js:97:5)
api_1        |     at async /node_modules/@nestjs/core/router/router-execution-context.js:45:28
api_1        |     at async /node_modules/@nestjs/core/router/router-proxy.js:8:17

Doesn't change imageOptions[] in test

Hello! I just tried your app and its very interesting and I found some issues. There is one of them:

In example we have const imageOptions = {} which I suppose should influence on browser config and env.
But if I changes this options in current test stays default settings and only text in web changes like this:

Maybe I do something wrong?

Baseline image is removed after removing build

Variation stays in DB and still references to missing image
Leeds to error 500 when trying to get baseline that does not exists anymore

Steps:

Accept test result
Remove all builds
Run test again

Error

api_1        | [Nest] 17   - 05/21/2020, 9:49:34 AM   [ExceptionsHandler] ENOENT: no such file or directory, open '/imageUploads/1590054492219.screenshot.png' +945529ms
api_1        | Error: ENOENT: no such file or directory, open '/imageUploads/1590054492219.screenshot.png'
api_1        |     at Object.openSync (fs.js:458:3)
api_1        |     at Object.readFileSync (fs.js:360:35)
api_1        |     at StaticService.getImage (/dist/shared/static/static.service.js:29:43)
api_1        |     at TestRunsService.create (/dist/test-runs/test-runs.service.js:97:49)
api_1        |     at TestService.postTestRun (/dist/test/test.service.js:29:52)
api_1        |     at processTicksAndRejections (internal/process/task_queues.js:97:5)
api_1        |     at async /node_modules/@nestjs/core/router/router-execution-context.js:45:28
api_1        |     at async /node_modules/@nestjs/core/router/router-proxy.js:8:17

Default user email is different from documentation

Log

migration_1  | ###########################
migration_1  | ## CREATING DEFAULT USER ##
migration_1  | ###########################
migration_1  | 
migration_1  | The user with the email "[email protected]" and password "123456" was created

actual [email protected]

expected [email protected]

name
width
height

https://github.com/Visual-Regression-Tracker/examples-java

After it's released:

api_1        | (node:17) UnhandledPromiseRejectionWarning: Error: 
api_1        | Invalid `prisma.user.findMany()` invocation in
api_1        | /dist/users/users.service.js:42:40
api_1        | 
api_1        | Error occurred during query execution:
api_1        | ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: Some(Error), code: SqlState("42P01"), message: "relation \"public.User\" does not exist", detail: None, hint: None, position: Some(Original(256)), where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("parse_relation.c"), line: Some(1173), routine: Some("parserOpenTable") }) }) })
api_1        |     at PrismaClientFetcher.request (/node_modules/@prisma/client/runtime/index.js:1:86042)
api_1        |     at processTicksAndRejections (internal/process/task_queues.js:97:5)
api_1        | (node:17) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
api_1        | (node:17) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
api_1        | (node:17) UnhandledPromiseRejectionWarning: Error: 
api_1        | Invalid `prisma.project.findMany()` invocation in
api_1        | /dist/projects/projects.service.js:37:43
api_1        | 
api_1        | Error occurred during query execution:
api_1        | ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(Error { kind: Db, cause: Some(DbError { severity: "ERROR", parsed_severity: Some(Error), code: SqlState("42P01"), message: "relation \"public.Project\" does not exist", detail: None, hint: None, position: Some(Original(128)), where_: None, schema: None, table: None, column: None, datatype: None, constraint: None, file: Some("parse_relation.c"), line: Some(1173), routine: Some("parserOpenTable") }) }) })
api_1        |     at PrismaClientFetcher.request (/node_modules/@prisma/client/runtime/index.js:1:86042)
api_1        |     at processTicksAndRejections (internal/process/task_queues.js:97:5)
api_1        | (node:17) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)

Temp workaround

restart containers

sometimes helps restart docker image

POC connection via socket

builds
test runs

to update:

https://github.com/Visual-Regression-Tracker/sdk-java

update README with new SDK and examples urls