d3ming/tagcompare

Image comparison for creative tags across browsers

╔╦╗╔═╗╔═╗  ╔═╗╔═╗╔╦╗╔═╗╔═╗╦═╗╔═╗
 ║ ╠═╣║ ╦  ║  ║ ║║║║╠═╝╠═╣╠╦╝║╣ 
 ╩ ╩ ╩╚═╝  ╚═╝╚═╝╩ ╩╩  ╩ ╩╩╚═╚═╝

There are many differences and issues when trying to render an HTML5 creative under different platform/browsers.

1. Fonts might look or be different
2. CSS / layout differences
3. Animation / timing differences

This tool tries to address the first 2 points above by doing image comparison of an HTML5 creative tag under different platform/browser configurations

Setup

Clone the repo: git clone [email protected]:paperg/tagcompare.git python setup.py install to install it. which tagcompare should show that it's installed

(optional): install phantomjs brew install phantomjs

settings.json

Make a local copy of settings.json called settings.local.json Update all the null values for saucelabs and placelocal:

  "saucelabs": {
    "user": null,
    "key": null
  },
  "placelocal": {
    "domain": "www.placelocaldemo.com",
    "secret": {
      "pl-secret": null,
      "pl-service-identifier": null
    }
  }

Update the settings.local.json file with the desired input to the tool as desired

Running the tool

tagcompare is the main entry point for the tool, it will capture and compare tags with the given input in your settings.local.json

Commandline arguments:

Commandline arguments can optionally be used to configure the tagcompare run. tagcompare --help will show the current list of supported options.

Note:Options passed through the commandline will override the settings.local.json

Outputs:

Outputs generated by the tool will be stored in /~/tagcompare

Running tests

make test or tox runs unit tests make test-all runs all the tests ./test <path_to_test> runs tests from a specific test file

tagcompare's Issues

Don't capture redundant screenshots

Compare existing screenshots before we capture new ones

Save info about saucelabs build alongside outputs

Add unit tests & test coverage

• Use pytest with the pytest-cov module
• Integrate with codeclimate

Refactor and make full test config matrix

Save builds for each compare run

Allow capture can be run separately from compare

Multi-thread browser config runs on SauceLabs

Optimize compare for text

Look into ways that we can optimize compare to distinguish differences in text, esp. line breaks

Add documentation to README

• Usage examples
• Input/output

Use logging framework / log levels instead of print statements

Log stats about number of images across different error thresholds

To help us figure out the right threshold - i.e.:

1000 images has error over 100
800 images has error over 200
300 images has error over 300
10 images has error over 1000

Make compares against 'golden' image better

'golden' is currently a regular config, which causes an extra capture run

We should ideally be able to not capture for golden, but have compare still work against it. Could do something like:

• Make 'golden' config enabled for compare but disabled for capture
• Have aggregate copy the 'chrome' config results to golden

Generate diff/merge images to debug image errors

Don't expose browserstack credential

Manage configurations in a config file

Comparisons always generate to default build

Normalize images before compare

Support running against multiple publications

Create better output structure on compare

• Potentially save very high diffs to a different folder
• Structure output such that you can easily trace tag HTML

Different compare threshold for different comparisons

For example - we want FF42 vs FF beta to have a lower threshold than a normal compare

Rename output.py and PathBuilder

A more appropriate name might be:

• job.py / Job
• result.py / Result

Restructure output

Restructure the output of the tool to:

output (main output dir):

• campaign (a directory for each campaign):
  • size (a directory for each size):
    • tag.html
    • chrome.png
    • firefox.png
    • ie9.png
    • comparison.png

Improve generated merged-images from compare

Ideas:

• Show many images side by side
• Always show compare against golden