reg-viz/reg-suit

:recycle: Visual Regression Testing tool

reg-suit's Issues

Create organization and move the ownership

I'll create some packages related to this tool, such as example repositories or dockerfiles. It's better that there's an organization to put them.

Add question to override conf when the target plugin was already configured

Now, prepare command override plugin configurations without user's permission.

Improvement build scripts

• Cache dependencies
• Store compiled packages between workflows

Create Icon

Remove dependency of `resolve` pkg

require.resolve is enough for me.

Write test code for core

add help and usage to CLI

ability to pass env value to plugins

use case:

• git-hash-plugin: base commit hash( or tag) to be expected key.
• notify-github-plugin: It needs the current commit hash. Usually, CI services export it such as CIRCLE_SHA1 or WERCKER_GIT_COMMIT

Error: Command failed: git rev-parse (detached from bedc554)

An error occurs from git-hash-plugin using in CI service (Wercker CI)

Error: Command failed: git rev-parse (detached from bedc554)
/bin/sh: 1: Syntax error: "(" unexpected

    at checkExecSyncError (child_process.js:588:13)
    at Object.execSync (child_process.js:628:13)
    at GitCmdClient.revParse (/pipeline/source/node_modules/reg-keygen-git-hash-plugin/lib/git-cmd-client.js:13:63)
    at CommitExplorer.getCurrentCommitHash (/pipeline/source/node_modules/reg-keygen-git-hash-plugin/lib/commit-explorer.js:17:35)
    at CommitExplorer.getBaseCommitHash (/pipeline/source/node_modules/reg-keygen-git-hash-plugin/lib/commit-explorer.js:23:30)
    at GitHashKeyGenPlugin.getExpectedKey (/pipeline/source/node_modules/reg-keygen-git-hash-plugin/lib/index.js:9:39)
    at RegProcessor.getExpectedKey (/pipeline/source/node_modules/reg-suit-core/lib/processor.js:24:39)
    at RegProcessor.runAll (/pipeline/source/node_modules/reg-suit-core/lib/processor.js:15:21)
    at Object.run [as default] (/pipeline/source/node_modules/reg-suit/lib/commands/run.js:6:22)
    at cli (/pipeline/source/node_modules/reg-suit/lib/cli.js:51:29)
    at Object.<anonymous> (/pipeline/source/node_modules/reg-suit/lib/cli.js:67:1)
    at Module._compile (module.js:569:30)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:503:32)
    at tryModuleLoad (module.js:466:12)
    at Function.Module._load (module.js:458:3)
    at Function.Module.runMain (module.js:605:10)
    at startup (bootstrap_node.js:158:16)
    at bootstrap_node.js:575:3

Missing to set threshold argument

https://github.com/reg-viz/reg-suit/blob/master/packages/reg-suit-core/src/processor.ts#L116
😱

Replace execSync('git ...') with library

• git-hash-plugin
• github-plugin

Add plugins

• s3 publisher
• key generator with env value (see #13)
• key generator with git hash
• GCS publisher ?
• Noop publisher ?
• slack(webhook) notifier
• github status notifier

Question interface

Now pluginPreparer's inquire is allowed to return a value any typed. Actually it's used with inquirer.prompt. So it should be compatible with inquirer.Question type.

Add ability indicate Progress / Spinner

I think it's useful such as logger.createSpinner() and logger.createProgress()

• https://www.npmjs.com/package/cli-spinner

• https://www.npmjs.com/package/cli-progress

• cli: install command

• s3-plugin download / upload / create-bucket?

• notify-github

• notify-slack

enabled/disabled flag to plugin specified config

Apply lint

• tslint
• editorconfig

Make gh-page responsive

prepare command creates configuration including abs path

Add pixel threshold

Please add pixel threshold option to reg-suit.
thresholdRate and thresholdPixel options have been added to reg-cli, and published as v0.11.0.(threshold option is alias to thresholdRate for now)

https://github.com/reg-viz/reg-cli/blob/master/src/index.js#L31-L33

Related pull Request

reg-viz/reg-cli#114

ability to register arbitrary subcommands from plugins

Use case:

• s3 publish
  • show the list already published
  • delete old reports

Putting report URL in PR comment(when succeed)

Comment to PR after CI running

Now notify-gh-plugin can comment to PR only if the associated PR is already open when reg-suit run. In other words it cannot comment if PR is open after CI running.

What's happen at TravisCI when PR from forked repo?

Hmmm, I think I'm in trouble detecting the previous and current git-hash....

The example job log is here. In this job, both previous(null) and current(1a95a49087dad68baec) hashes are incorrect.

The above job was triggered PR Quramy/rs-fork-pr-demo-repo#2, which into origin/master from quramy-alt:master. Now, to detection base-hash, the CI script(.travis.yml) executes the following commands:

before_script:
  - git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
  - git fetch origin
  - git checkout $TRAVIS_BRANCH || git checkout -b $TRAVIS_BRANCH

The above workaround looked good for us only if PRs are limited in-house, but when the PR is derived from other(forked) repos, it does not make sense. It's because that the fetched origin doesn't have the forked repository. This command updates from not forked but source repository.

And more nastily, if the env value $TRAVIS_BRANCH is equal to "master" the checkout $TRAVIS_BRANCH exit with 0 successfully because the pointed branch is exists in both forked and source repos.

reg-keygen-git-hash-plugin timeout

reg-keygen-git-hash-plugin hangs up(timeout on circleCI) after warning message. (e.g. warning: ignoring origin/xxxx; cannot handle more than 25 refs).

keygen-git-hash-plugin sometimes returns incorrect commit

Repro:

mkdir -p git-hash-test
cd git-hash-test
git init
git commit --allow-empty -m "first commit"
git checkout -b feat-x
git commit --allow-empty -m "x1"
git commit --allow-empty -m "x2"
git checkout master
git commit --allow-empty -m "master1"
git commit --allow-empty -m "master2"
git checkout -b master2x feat-x
git merge master

And git show-branch -a says the following:

! [feat-x] x2
 ! [master] master2
  * [master2x] x2
---
 +  [3e4fb34] master2
 +  [83e1ee9] master1
+ * [5b77826] x2
+ * [6d28d6c] x1
++* [efdf06c] first commit

Expected behaviour : keygen-git-hash plugin detects 5b77826(x2) commit as the previous key.
Actual behaviour: keygen-git-hash plugin fail to detect or returns 3e4fb34 (master2) commit as the previous key.

restructure core parameters

I want better param names:

• workingDir
• actualDir. Now it's a relative from workingDir but it should be independent workingDir (see #26)
• remove key expectedDir ?. it should be pointed by fetch plugin.
• threshold

Add contributing guide

• How to build
• How to test

Separate runner and preparer from core.ts

Enable chalk under CI environment

reg-cli output are with colors, however reg-suit-cli output ain't. why?

Handle error when AWS credentials are not found

better plugin interface

I think there are better names of each plugins and their methods.

Publisher:

Publisher plugin should have 2 methods named publish and fetch. They've been strongly influenced by the S3-plugin implementation and I feel uneasy.

• What does means fetch ? Some users wants this method to copy local files or refer a directory existing their repository.
• Should the fetch method be tied with the publish ? In the beginning I've not doubted that the location publish to / fetch from is same, is it correct ?

Key Generator

KeyGenerator has strongly tied 2 methods too, getExpectedKey and getActualKey.

Because the above reason, in other words I think separate publish / fetch to other interfaces, getExpectedKey and getActualKey should be declared at other interfaces rather than one interface.

The second, I think the term key and generate ain't good, but I cannot get better name than them... Is there nice one?

• identifier / key / snapshot-id / snapshot-key / etc,,,
• detect / create / determine / find / search / etc,,,

Display expected / actual key in PR comment body

Motivation

When I confirm the expected key calculated by keygen-git-plugin, it's too inconvenient to visit the CI log page and pick the reg-suit log line up...

Write blog post

• Medium(en-us)
• Qiita(ja-jp)

`prepare -p` clear other plugins configs

after -p notify-slack, configured other plugins' options are overridden by empty...

Separate the `rev` option from git hash plugin

Deploy script has broken

😱 It seems that failure of deploying serverless does not stop CI process. And finally github-plugin packages misses the API endpoint which should be generated by successful serverless deploy.

For now, please exec lerna publish at your local machine...

deploy gh-app API with CI

Append working dir into gitignore when initialization

Error: Cannot find module occurred

If known or unnecessary, please close this issue 🙏

version: v0.017

Error: Cannot find module occurred when I try to initialize reg-notify-github-plugin with init command.

{ Error: Cannot find module 'reg-keygen-git-hash-plugin/lib/commit-explorer'
    at Function.Module._resolveFilename (module.js:470:15)
    at Function.Module._load (module.js:418:25)
    at Module.require (module.js:498:17)
    at require (internal/module.js:20:19)
    at Object.<anonymous> (/Users/username/sample/node_modules/reg-notify-github-plugin/lib/github-notifier-plugin.js:3:27)
    at Module._compile (module.js:571:32)
    at Object.Module._extensions..js (module.js:580:10)
    at Module.load (module.js:488:32)
    at tryModuleLoad (module.js:447:12)
    at Function.Module._load (module.js:439:3) code: 'MODULE_NOT_FOUND' }

actual image files are not published when `core.actual` points outside working dir.

reproducing it with the following conf :

{
  "core": {
    "workingDir": "reg",
    "actualDir": "../viz-test",
    "expectedDir": "expected"
  },
  "plugins": {
    "reg-keygen-git-hash-plugin": true,
    "reg-notify-github-plugin": {
      "installationId": "37779",
      "owner": "Quramy",
      "repository": "reg-simple-demo",
      "prComment": true
    },
    "reg-publish-s3-plugin": {
      "bucketName": "reg-publish-bucket-14ccadad-40b7-45b2-8d47-e278a849aa3b"
    }
  }
 }

GH app pr-comment API returns 502 when associated PR is not found

Plugin preset

I want to point the recommended plugins when user's init. Can I use inquirer default value, something like:

✅ keygen-git-base-hash-plugin
⬛️ notify-slack-plugin

install plugin

Using fixed commit hash or tag name as the expected key

Proposal:

Add new reg-keygen-git-hash-plugin opetion expected.type.

"plugins": {
  "reg-keygen-git-hash": {
      "expectedType": "revision",
      "expectedRevision": "v1.0.0"
  }
}

And if using base-hash pattern, config follows:

"plugins": {
  "reg-keygen-git-hash": {
      "expectedType": "base-commit"
  }
}

Enhance logger

• color interface
• simple implementation for testing plugins
• display category
• object dump for verbose

Handle error when incoming webhook URL is not found

Better configuration for notify-gh-plugin

Now notify-gh-plugin requires the following parameter:

• Owner name
• Repository name
• Installation ID

I think it's redundant. Is there a better way to identify installed repository? If I could get a list user's installations with v3/v4 API... but I can't find one.

Dcumentation

• GitHub pages
• README
  • Plugin summaries
  • Guide for CI, example link
  • Core configuration
• Plugins
  • keygen-git-hash
    • What plugin does
  • notify-slack
    • Install Guide( requirements webhook)
    • Configuration type
  • notify-github
    • What plugin does
    • Install Guide
    • Configuration type
  • publish-s3
    • What plugin does
    • Install Guide (AWS credentials)
    • Configuration type

Upgrade TypeScrpt to the latest version

fetch / publish command