terriblelabs/motion-juxtapose

Screenshot-driven assertions for testing Rails and RubyMotion applications.

Juxtapose

Screenshot-driven assertions for testing Rails and RubyMotion applications.

Installation

Motion-juxtapose uses ImageMagick under the hood for image comparison. If you're using Homebrew, you can install it with:

brew install imagemagick

Add this line to your application's Gemfile:

gem 'motion-juxtapose'

And then execute:

$ bundle

Or install it yourself as:

$ gem install motion-juxtapose

Usage

Juxtapose's workflow generally goes:

1. Write a test that puts your app in the state you want to capture and take a screenshot (see below for how to do this in Bacon and Frank).
2. The first time this screenshot is captured, your test will fail with the error "No accepted screen shot".
3. You must provide Juxtapose an accepted screenshot to match against in future test runs. Run the juxtapose server with bundle exec juxtapose and browse to http://localhost:4567 — you should see the just-captured screenshot, like below.
4. If the screenshot looks like what you expect, click Accept to save the image. Future runs of this test will be compared against the accepted one.
5. Run the tests again, and they should be green.
6. Later, when you introduce a change (intentionally or not), the tests will fail with the message "Screenshot did not match". Open up the juxtapose server again, and you'll see a comparison of your accepted screenshot with what the test captured.
7. If the change was intentional, you can accept the new screenshot - otherwise, you've got some fixin' to do!

Testing RubyMotion Apps

MacBacon

Juxtapose provides a single new matcher: it_should_look_like takes a single argument, a descriptive string predicate of what the screen should look like.

Example:

describe "home controller" do
  extend Juxtapose

  describe "home controller" do
    tests HomeViewController

    it "has a button that moves when tapped" do
      tap "Goodbye, world!"
      it_should_look_like "when_tapped"
    end
  end
end

Frank

Add the following config to your features/support/env.rb:

require 'motion-juxtapose'
Juxtapose::FrankStrategy.setup

This lets you write a screenshot matcher along the lines of:

Then /^the screen should match "([^\"]*)"$/ do |template|
  wait_for_nothing_to_be_animating
  screenshotter = Juxtapose::Screenshotter.new(self, template)
  max_attempts = 20
  expect(screenshotter.attempt_verify(max_attempts)).to eq(true)
end

Appium

Add the following config to your features/support/env.rb:

require 'motion-juxtapose'
Juxtapose::AppiumStrategy.setup

This lets you write a screenshot matcher along the lines of:

Then /^the screen should match "([^\"]*)"$/ do |template|
  wait_for_nothing_to_be_animating
  #unlike frank we need to pass in a project root to find the screenshot directory
  screenshotter = Juxtapose::Screenshotter.new(self, template, 0, File.expand_path('./'))

  max_attempts = 20
  expect(screenshotter.attempt_verify(max_attempts)).to eq(true)
end

Testing Rails Apps

Capybara

Any Capybara driver that supports screenshot capture should work, but I've only tested this with Poltergeist so far.

In your spec_helper.rb, add:

require 'juxtapose/capybara'

# if you're using Rspec, you can get the `look_like?` custom matcher with:
require 'juxtapose/rspec'

Now the Capybara page object will have a looks_like?(predicate) method that can call to make screenshot assertions:

feature "viewing locations", js: true do
  scenario "should only show books at first location" do
    visit location_path Location.find_by_name("Cambridge")
    expect(page).to look_like("books at Cambridge")
  end
end

feature "viewing locations with a fuzz factor", js: true do
  scenario "should only show books at first location" do
    visit location_path Location.find_by_name("Boston", fuzz_factor: 4)
    expect(page).to look_like("books at Boston")
  end
end

Fuzzy matching

Sometimes screenshots will differ slightly due to things like animation timing. Juxtapose will try to match an existing screenshot multiple times to help counteract this, but it can still be useful to be able to match screenshots using a percent similarity threshold.

In bacon specs, you can pass a fuzz factor from 0 to 100 to it_should_look_like, 0 meaning an exact match and 100 matching anything of the same size. The default is 0.

# matches images that are 15% similar
it_should_look_like 'a fancy animation', 15

In Frank, pass the fuzz factor in as the third argument to the Screenshotter constructor. A more exacting Frank step could be written as:

Then /^the screen should match "([^\"]*)" within (\d+)% in (\d+) attempts$/ do |template, fuzz_factor, max_attempts|
  wait_for_nothing_to_be_animating
  screenshotter = Juxtapose::Screenshotter.new(self, template, fuzz_factor)
  expect(screenshotter.attempt_verify(max_attempts)).to eq(true)
end

Juxtapose Server

Juxtapose comes with a small webapp that you can use to view your screenshot specs, see diffs between accepted and failing specs and accept any changed images that are expected changes.

To start it, run bundle exec juxtapose in the root of your project and browse to localhost:4567.

Release Notes

v.0.4.0

• Updates for RubyMotion 6.1 & iOS 12.2 (@wndxlori)
• Fixes some undefined warnings (RSpec/Capybara/Cucumber)
• Replaces renderInContext with drawViewHierarchyInRect
• Device names are no longer calculated from screen sizes, but from UIDevice.currentDevice

v.0.3.2

• Fixes bug where rails projects could not use fuzz factor (@squidpunch)

v.0.3.1

• Fixes bug where you could not test against non-portrait screenshots (@ohammersmith)

v.0.3.0

• Add Appium support (@squidpunch)
• Add support for 6/6+ screen sizes (@squidpunch)

v.0.2.0

• Rails/rspec support
• Fixes to work under newest version of ImageMagick

v.0.1.1

• Raise error if screenshot sizes don't match (@mdenomy)
• Bug fix: Differentiate image status by filename, not full path. Fixes #8.

v.0.1.0

• Initial release.

Contributors

• Joe Lind
• Thomas Mayfield
• Jeffrey Chupp
• Michael Denomy
• David Larrabee
• Otto Hammersmith
• Lori Olson

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

motion-juxtapose's Issues

Failures when simulator is in non-portrait orientation

If I rotate the device before taking a screenshot, I get this sort of error:

  - looks like demo screen [ERROR: NameError - uninitialized constant Juxtapose::MacBaconStrategy::M_PI_2]

NameError: uninitialized constant Juxtapose::MacBaconStrategy::M_PI_2
    mac_bacon_strategy.rb:47:in `save_current:': DemoScreen - looks like demo screen
    screenshotter.rb:117:in `verify'
    screenshotter.rb:107:in `attempt_verify:'
    screenshotter.rb:23:in `looks_like?:'
    screenshotter.rb:27:in `it_should_look_like:'
    spec.rb:459:in `execute_block'
    spec.rb:421:in `run_postponed_block:’

The issue is that the constants for π set in screens hotter.rb aren't picked up over in mac_bacon_strategy.rb.

I have a fix and will create a PR/patch shortly.

In Rails/Ember.js app, screenshots are not fully loaded

The app that I'm trying to use this on is a Rails 4.2 app with an Ember.js front-end. We use Cucumber, Capybara, Poltergeist, and capybara-ember (to solve some of the async waiting issues that Capybara can't handle on its own).

When using the app in a web browser, it looks good, but when the test creates the screenshot, it looks wrong. The screenshots look like the page hasn't finished loading, but I've added sleep statements before generating the screenshot, and they look the same.

Example 1: Notice how the green background doesn't span the full window and the submit button is too small.

Example 2: However in this example the same screen renders the full size of the window, but the submit button is still rendered too small:

Example 3: Screenshot appears to be in the middle of rendering the orange background. I intentionally blurred the logo.

I don't know if this has anything to do with Ember's async routing and rendering system. Could be related to the same issues we've been having (which required the use of capybara-ember to sleep until the async routing/rendering completed).

Add support for web-front end examining multiple directories of screenshots

Screenshots are different sizes (even though they aren't)

I'm getting a "screenshots are different sizes" even though they're clearly the same.

My spec:

describe LoginScreen do
  extend Juxtapose
  tests LoginScreen

  def login_screen
    @login_screen ||= LoginScreen.new
  end

  def controller
    login_screen
  end

  after { @login_screen = nil }

  it "has a logo, email field, password field, submit button, and forgot password button" do
    view("Sign In").should.be.kind_of(UILabel)
    it_should_look_like "login_screen", 4
  end
end

Show changed screenshots since master

Or perhaps since an arbitrary branch/commit.

This would be useful when you've accepted a lot of new screenshots and want to see a more holistic view of what's changed.

Improve error messages

"false != true" is not very friendly.

Cleanup task to remove all screenshots not currently used

Add Frank steps that can be imported instead of just an example

Two identical screenshots not being matched

The two screenshots are identical. I have a 4% "fuzz factor". Running the command on the terminal appears to work fine.

compare -fuzz 4% -metric AE -dissimilarity-threshold 1 -subimage-search accepted.png current.png diff.png
0 @ 0,0%

Is motion-juxtapose expecting a different output?

Scroll bar causing intermittent failure

Hey, love the gem. One thing: a scroll bar is appearing on a UITableView and causing the spec to fail.

If I accept the change and re-run, half the time it'll change again.

Maybe some sort of % tolerance, like motion-screenspecs?

Support for device_name

Running motion-juxtapose with a different device name (say, rake spec device_name='iPhone 6 Plus') causes failures:

ERROR: RuntimeError - Screenshots are different sizes
# RuntimeError: Screenshots are different sizes

Escape certain characters in the file path

Using a screenshot name of "name (bad)" resulted in some printed statements when I ran my tests.

sh: -c: line 0: syntax error near unexpected token (' sh: -c: line 0:mkdir -p [project_path]/spec/screens/capybara/web/spec-description/name-(bad)'

handle mismatched screenshot sizes gracefully

Right now compare hangs if you're comparing screenshots of different sizes.

Fail loudly if ImageMagick is not installed

And add documentation that it's required.

Don't require ios_version to be implemented in app_delegate

Rails gem like this?

I like motion-juxtapose a lot. Is there a Rails (or Ruby) equivalent we could use on Rails web apps?

Capybara fuzzy matching

There are examples of how to do fuzzy matching in Bacon and Frank, but I can't figure out how to do it for a rails app using capybara.

I tried expect(page).to look_like("Profile", 15) with no success. Any help would be much appreciated

Thanks!

If your screenshot name has the word "current" in it, juxtapose's webui gets confused.

It ends up showing the accepted screenshot for both Accepted and Current. Looks like the matcher needs to be smarter here.