testdouble/suture

🏥 A Ruby gem that helps you refactor your legacy code

Suture 🏥

A refactoring tool for Ruby, designed to make it safe to change code you don't confidently understand. In fact, changing untrustworthy code is so fraught, Suture hopes to make it safer to completely reimplement a code path.

Suture provides help to the entire lifecycle of refactoring poorly-understood code, from local development, to a staging environment, and even in production.

Video

Suture was unveiled at Ruby Kaigi 2016 as one approach that can make refactors less scary and more predictable. You can watch the 45 minute screencast here:

Walk-through guide

Refactoring or reimplementing important code is an involved process! Instead of listing out Suture's API without sufficient exposition, here is an example that will take you through each stage of the lifecycle.

Development

Suppose you have a really nasty worker method:

class MyWorker
  def do_work(id)
    thing = Thing.find(id)
    # … 99 lines of terribleness …
    MyMailer.send(thing.result)
  end
end

1. Identify a seam

A seam serves as an artificial entry point that sets a boundary around the code you'd like to change. A good seam is:

• easy to invoke in isolation
• takes arguments, returns a value
• eliminates (or at least minimizes) side effects (for more on side effects, see this tutorial)

Then, to create a seam, typically we create a new unit to house the code that we excise from its original site, and then we call it. This adds a level of indirection, which gives us the flexibility we'll need later.

In this case, to create a seam, we might start with this:

class MyWorker
  def do_work(id)
    MyMailer.send(LegacyWorker.new.call(id))
  end
end

class LegacyWorker
  def call(id)
    thing = Thing.find(id)
    # … Still 99 lines. Still terrible …
    thing.result
  end
end

As you can see, the call to MyMailer.send is left at the original call site. MyMailer.send is effectively a void method being invoked for its side effect, which would make it difficult to test. By creating LegacyWorker#call, we can now express the work more clearly in terms of repeatable inputs (id) and outputs (thing.result), which will help us verify that our refactor is working later.

Since any changes to the code while it's untested are very dangerous, it's important to minimize changes made for the sake of creating a clear seam.

2. Create our seam

Next, we introduce Suture to the call site so we can start analyzing its behavior:

class MyWorker
  def do_work(id)
    MyMailer.send(Suture.create(:worker, {
      old: LegacyWorker.new,
      args: [id]
    }))
  end
end

Where old can be anything callable with call (like the class above, a method, or a Proc/lambda) and args is an array of the args to pass to it.

At this point, running this code will result in Suture just delegating to LegacyWorker without taking any other meaningful action.

3. Record the current behavior

Next, we want to start observing how the legacy worker is actually called. What arguments are being sent to it and what value does it returns (or, what error does it raise)? By recording the calls as we use our app locally, we can later test that the old and new implementations behave the same way.

First, we tell Suture to start recording calls by setting the environment variable SUTURE_RECORD_CALLS to something truthy (e.g. SUTURE_RECORD_CALLS=true bundle exec rails s). So long as this variable is set, any calls to our seam will record the arguments passed to the legacy code path and the return value.

As you use the application (whether it's a queue system, a web app, or a CLI), the calls will be saved to a sqlite database. Keep in mind that if the legacy code path relies on external data sources or services, your recorded inputs and outputs will rely on them as well. You may want to narrow the scope of your seam accordingly (e.g. to receive an object as an argument instead of a database id).

Hard to exploratory test the code locally?

If it's difficult to generate realistic usage locally, then consider running this step in production and fetching the sqlite DB after you've generated enough inputs and outputs to be confident you've covered most realistic uses. Keep in mind that this approach means your test environment will probably need access to the same data stores as the environment that made the recording, which may not be feasible or appropriate in many cases.

4. Ensure current behavior with a test

Next, we should probably write a test that will ensure our new implementation will continue to behave like the old one. We can use these recordings to help us automate some of the drudgery typically associated with writing characterization tests.

We could write a test like this:

class MyWorkerCharacterizationTest < Minitest::Test
  def setup
    super
    # Load the test data needed to resemble the environment when recording
  end

  def test_that_it_still_works
    Suture.verify(:worker, {
      :subject => LegacyWorker.new
      :fail_fast => true
    })
  end
end

Suture.verify will fail if any of the recorded arguments don't return the expected value. It's a good idea to run this against the legacy code first, for two reasons:

• running the characterization tests against the legacy code path will ensure that the test environment has the data needed to behave the same way as when it was recorded (it may be appropriate to take a snapshot of the database before you start recording and load it before you run your tests)

• by generating a code coverage report (simplecov is a good one to start with) from running this test in isolation, we can see what LegacyWorker is actually calling, in an attempt to do two things:

  • maximize coverage for code in the LegacyWorker (and for code that's subordinate to it) to make sure our characterization test sufficiently exercises it
  • identify incidental coverage of code paths that are outside the scope of what we hope to refactor. This will help to see if LegacyWorker has side effects we didn't anticipate and should additionally write tests for

5. Specify and test a path for new code

Once the automated characterization test of our recordings is passing, then we can start work on a NewWorker. To get started, we update our Suture configuration:

class MyWorker
  def do_work(id)
    MyMailer.send(Suture.create(:worker, {
      old: LegacyWorker.new,
      new: NewWorker.new,
      args: [id]
    }))
  end
end

class NewWorker
  def call(id)
  end
end

Next, we specify a NewWorker under the :new key. For now, Suture will start sending all of its calls to NewWorker#call.

Next, let's write a test to verify the new code path also passes the recorded interactions:

class MyWorkerCharacterizationTest < Minitest::Test
  def setup
    super
    # Load the test data needed to resemble the environment when recording
  end

  def test_that_it_still_works
    Suture.verify(:worker, {
      subject: LegacyWorker.new,
      fail_fast: true
    })
  end

  def test_new_thing_also_works
    Suture.verify(:worker, {
      subject: NewWorker.new,
      fail_fast: false
    })
  end
end

Obviously, this should fail until NewWorker's implementation covers all the cases that we recorded from LegacyWorker.

Remember, characterization tests aren't designed to be kept around forever. Once you're confident that the new implementation is sufficient, it's typically better to discard them and design focused, intention-revealing tests for the new implementation and its component parts.

6. Refactor or reimplement the legacy code.

This step is the hardest part and there's not much Suture can do to make it any easier. How you go about implementing your improvements depends on whether you intend to rewrite the legacy code path or refactor it. Some comments on each approach follow:

Reimplementing

The best time to rewrite a piece of software is when you have a better understanding of the real-world process that it models than the original authors did when they first wrote it. If that's the case, it's likely you'll think of more reliable names and abstractions than they did.

As for workflow, consider writing the new implementation like you would any other new part of the system. The added benefit is being able to run the characterization tests as a progress indicator and a backstop for any missed edge cases. The ultimate goal of this workflow should be to incrementally arrive at a clean design that completely passes the characterization test run by running Suture.verify.

Refactoring

If you choose to refactor the working implementation, though, you should start by copying it (and all of its subordinate types) into the new, separate code path. The goal should be to keep the legacy code path in a working state, so that Suture can run it when needed until we're supremely confident that it can be safely discarded. (It's also nice to be able to perform side-by-side comparisons without having to check out a different git reference.)

The workflow when refactoring should be to take small, safe steps using well understood refactoring patterns and running the characterization test suite frequently to ensure nothing was accidentally broken.

Once the code is factored well enough to work with (i.e. it is clear enough to incorporate future anticipated changes), consider writing some clear and clean unit tests around new units that shook out from the activity. Having good tests for well-factored code is the best guard against seeing it slip once again into poorly-understood "legacy" code.

Staging

Once you've changed the code, you still may not be confident enough to delete it entirely. It's possible (even likely) that your local exploratory testing didn't exercise every branch in the original code with the full range of potential arguments and broader state.

Suture gives users a way to experiment with risky refactors by deploying them to a staging environment and running both the original and new code paths side-by-side, raising an error in the event they don't return the same value. This is governed by the :call_both to true:

class MyWorker
  def do_work(id)
    MyMailer.send(Suture.create(:worker, {
      old: LegacyWorker.new,
      new: NewWorker.new,
      args: [id],
      call_both: true
    }))
  end
end

With this setting, the seam will call through to both legacy and refactored implementations, and will raise an error if they don't return the same value. Obviously, this setting is only helpful if the paths don't trigger major or destructive side effects.

Production

You're almost ready to delete the old code path and switch production over to the new one, but fear lingers: maybe there's an edge case your testing to this point hasn't caught.

Suture was written to minimize the inhibition to moving forward with changing code, so it provides a couple features designed to be run in production when you're yet unsure that your refactor or reimplementation is complete.

Logging errors

While your application's logs aren't affected by Suture, it may be helpful for Suture to maintain a separate log file for any errors that are raised by the refactored code path.

Suture has a handful of process-wide logging settings that can be set at any point as your app starts up (if you're using Rails, then your environment-specific (e.g. config/environments/production.rb) config file is a good choice).

Suture.config({
  :log_level => "WARN", #<-- defaults to "INFO"
  :log_stdout => false, #<-- defaults to true
  :log_io => StringIO.new,      #<-- defaults to nil
  :log_file => "log/suture.log" #<-- defaults to nil
})

When your new code path raises an error with the above settings, it will propagate and log the error to the specified file.

Custom error handlers

Additionally, you may have some idea of what you want to do (i.e. phone home to a reporting service) in the event that your new code path fails. To add custom error handling before, set the :on_error option to a callable.

class MyWorker
  def do_work(id)
    MyMailer.send(Suture.create(:worker, {
      old: LegacyWorker.new,
      new: NewWorker.new,
      args: [id],
      on_error: -> (name, args) { PhonesHome.new.phone(name, args) }
    }))
  end
end

Retrying failures

Since the legacy code path hasn't been deleted yet, there's no reason to leave users hanging if the new code path explodes. By setting the :fallback_on_error entry to true, Suture will rescue any errors raised from the new code path and attempt to invoke the legacy code path instead.

class MyWorker
  def do_work(id)
    MyMailer.send(Suture.create(:worker, {
      old: LegacyWorker.new,
      new: NewWorker.new,
      args: [id],
      fallback_on_error: true
    }))
  end
end

Since this approach rescues errors, it's possible that errors in the new code path will go unnoticed, so it's best used in conjunction with Suture's logging feature. Before ultimately deciding to finally delete the legacy code path, double-check that the logs aren't full of rescued errors!

Public API Summary

• Suture.create(name, opts) - Creates a seam in your production source code
• Suture.verify(name, opts) - Verifies a callable subject can recreate recorded calls
• Suture.config(config) - Sets logging options, as well global defaults for other properties
• Suture.reset! - Resets all Suture configuration
• Suture.delete!(id) - Deletes a recorded call by id
• Suture.delete_all!(name) - Deletes all recorded calls for a given seam name

Configuration

Legacy code is, necessarily, complex and hard-to-wrangle. That's why Suture comes with a bunch of configuration options to modify its behavior, particularly for hard-to-compare objects.

Setting configuration options

In general, most configuration options can be set in several places:

• Globally, via an environment variable. The flag record_calls will translate to an expected ENV var named SUTURE_RECORD_CALLS and can be set from the command line like so: SUTURE_RECORD_CALLS=true bundle exec rails server, to tell Suture to record all your interactions with your seams without touching the source code. (Note: this is really only appropriate if your codebase only has one Suture seam in progress at a time, since using a global env var configuration for one seam's sake will erroneously impact the other.)

• Globally, via the top-level Suture.config method. Most variables can be set via this top-level configuration, like Suture.config(:database_path => 'my.db'). Once set, this will apply to all your interactions with Suture for the life of the process until you call Suture.reset!.

• At a Suture.create or Suture.verify call-site as part of its options hash. If you have several seams, you'll probably want to set most options locally where you call Suture, like Suture.create(:foo, { :comparator => my_thing })

Supported options

Suture.create

Suture.create(name, [options hash])

• name (Required) - a unique name for the seam, by which any recordings will be identified. This should match the name used for any calls to Suture.verify by your automated tests

• old - (Required) - something that responds to call for the provided args of the seam and either is the legacy code path (e.g. OldCode.new.method(:old_path)) or invokes it (inside an anonymous Proc or lambda)

• args - (Required) - an array of arguments to be passed to the old or new

• new - like old, but either references or invokes the code path designed to replace the old legacy code path. When set, Suture will default to invoking the new path at the exclusion of the old path (unless a mode flag like record_calls, call_both, or fallback_on_error suggests differently)

• database_path - (Default: "db/suture.sqlite3") - a path relative to the current working directory to the Sqlite3 database Suture uses to record and playback calls

• record_calls - (Default: false) - when set to true, the old path is called (regardless of whether new is set) and its arguments and result (be it a return value or an expected raised error) is recorded into the Suture database for the purpose of more coverage for calls to Suture.verify. Read more

• call_both - (Default: false) - when set to true, the new path is invoked, then the old path is invoked, each with the seam's args. The return value from each is compared with the comparator, and if they are not equivalent, then a Suture::Error::ResultMismatch is raised. Intended after the new path is initially developed and to be run in pre-production environments. Read more

• fallback_on_error - (Default: false) - designed to be run in production after the initial development of the new code path, when set to true, Suture will invoke the new code path. If new raises an error that isn't an expected_error_type, then Suture will invoke the old path with the same args in an attempt to recover a working state for the user. Read more

• raise_on_result_mismatch - (Default: true) - when set to true, the call_both mode will merely log incidents of result mismatches, as opposed to raising Suture::Error::ResultMismatch

• return_old_on_result_mismatch - (Default: false) - when set to true, the call_both mode will return the result of the old code path instead of the new code path. This is useful when you want to log mismatches in production (i.e. when you're very confident it's safe and fast enough to use call_both in production), but want to fallback to the old path in the case of a mismatch to minimize disruption to your users

• comparator - (Default: Suture::Comparator.new) - determines how return values from the Suture are compared when invoking Suture.verify or when call_both mode is activated. By default, results will be considered equivalent if == returns true or if they Marshal.dump to the same string. If this default isn't appropriate for the return value of your seam, read on

• expected_error_types - (Default: []) - if the seam is expected to raise certain types of errors, don't consider them to be exceptional cases. For example, if your :widget seam is known to raise WidgetError objects in certain cases, setting :expected_error_types => [WidgetError] will result in:

  • Suture.create will record expected errors when record_calls is enabled
  • Suture.verify will compare recorded and actual raised errors that are kind_of? any recorded error type (regardless of whether Suture.verify is passed a redundant list of expected_error_types)
  • Suture.create, when fallback_on_error is enabled, will allow expected errors raised by the new path to propagate, as opposed to logging & rescuing them before calling the old path as a fallback

• disable - (Default: false) - when enabled, Suture will attempt to revert to the original behavior of the old path and take no special action. Useful in cases where a bug is discovered in a deployed environment and you simply want to hit the brakes on any new code path experiments by setting SUTURE_DISABLE=true globally

• dup_args - (Default: false) - when enabled, Suture will call dup on each of the args passed to the old and/or new code paths. Useful when the code path(s) mutate the arguments in such a way as to prevent call_both or fallback_on_error from being effective

• after_new - a call-able hook that runs after new is invoked. If new raises an error, it is not invoked

• after_old - a call-able hook that runs after old is invoked. If old raises an error, it is not invoked

• on_new_error - a call-able hook that is invoked after new raises an unexpected error (see expected_error_types).

• on_old_error - a call-able hook that is invoked after old raises an unexpected error (see expected_error_types).

Suture.verify

Suture.verify(name, [options hash])

Many of the settings for Suture.verify mirror the settings available to Suture.create. In general, the two methods' common options should be configured identically for a given seam; this is necessary, because the Suture.verify call site doesn't depend on (or know about) any Suture.create call site of the same name; the only resource they share is the recorded calls in Suture's database.

• name - (Required) - should be the same name as a seam for which some number of recorded calls exist

• subject - (Required) - a call-able that will be invoked with each recorded set of args and have its result compared to that of each recording. This is used in lieu of old or new, since the subject of a Suture.verify test might be either (or neither!)

• database_path - (Default: "db/suture.sqlite3") - as with Suture.create, a custom database path can be set for almost any invocation of Suture, and Suture.verify is no exception

• verify_only - (Default: nil) - when set to an ID, Suture.verify` will only run against recorded calls for the matching ID. This option is meant to be used to focus work on resolving a single verification failure

• fail_fast - (Default: false) - Suture.verify will, by default, run against every single recording, aggregating and reporting on all errors (just like, say, RSpec or Minitest would). However, if the seam is slow to invoke or if you confidently expect all of the recordings to pass verification, fail_fast is an appropriate option to set.

• call_limit - (Default: nil) - when set to a number, Suture will only verify up to the set number of recorded calls. Because Suture randomizes the order of verifications by default, you can see this as setting Suture.verify to sample a random smattering of call_limit recordings as a smell test. Potentially useful when a seam is very slow

• time_limit - (Default: nil) - when set to a number (in seconds), Suture will stop running verifications against recordings once time_limit seconds has elapsed. Useful when a seam is very slow to invoke

• error_message_limit - (Default: nil) - when set to a number, Suture will only print up to error_message_limit failure messages. That way, if you currently have hundreds of verifications failing, your console isn't overwhelmed by them on each run of Suture.verify

• random_seed - (Default: it's random!) - a randomized seed used to shuffle the recordings before verifying them against the subject code path. If set to nil, the recordings will be invoked in insertion-order. If set to a specific number, that number will be used as the random seed (useful when re-running a particular verification failure that can't be reproduced otherwise)

• comparator - (Default: Suture::Comparator) - If a custom comparator is used by the seam in Suture.create, then the same comparator should probably be used by Suture.verify to ensure the results are comparable. Read more on creating custom comparators )

• expected_error_types - (Default: []) - this option has little impact on Suture.verify (since each recording will either verify a return value or an error in its own right), however it can be set to squelch log messages warning that errors were raised when invoking the subject

• after_subject - a call-able hook that runs after subject is invoked. If subject raises an error, it is not invoked

• on_new_subject - a call-able hook that is invoked after subject raises an unexpected error (see expected_error_types)

Creating a custom comparator

Out-of-the-box, Suture will do its best to compare your recorded & actual results to ensure that things are equivalent to one another, but reality is often less tidy than a gem can predict up-front. When the built-in equivalency comparator fails you, you can define a custom one—globally or at each Suture.create or Suture.verify call-site.

Extending the built-in comparator class

If you have a bunch of value types that require special equivalency checks, it makes sense to invest the time to extend built-in one:

class MyComparator < Suture::Comparator
  def call(recorded, actual)
    if recorded.kind_of?(MyType)
      recorded.data_stuff == actual.data_stuff
    else
      super
    end
  end
end

So long as you return super for non-special cases, it should be safe to set an instance of your custom comparator globally for the life of the process with:

Suture.config({
  :comparator => MyComparator.new
})

Creating a one-off comparator

If a particular seam requires a custom comparator and will always return sufficiently homogeneous types, it may be good enough to set a custom comparator inline at the Suture.create or Suture.verify call-site, like so:

Suture.create(:my_type, {
  :old => method(:old_method),
  :args => [42],
  :comparator => ->(recorded, actual){ recorded.data_thing == actual.data_thing }
})

Just be sure to set it the same way if you want Suture.verify to be able to test your recorded values!

Suture.verify(:my_type, {
  :subject => method(:old_method),
  :comparator => ->(recorded, actual){ recorded.data_thing == actual.data_thing }
})

Comparing two ActiveRecord objects

Let's face it, a massive proportion of legacy Ruby code in the wild involves ActiveRecord objects to some extent, and it's important that Suture be equipped to compare them gracefully. If Suture's default comparator (Suture::Comparator) detects two ActiveRecord model instances being compared, it will behave differently, by this logic:

1. Instead of comparing the objects with == (which returns true so long as the id attribute matches), Suture will compare the objects' attributes hashes instead
2. The built-in updated_at and created_at will typically differ when code is executed at different times and are usually not meaningful to application logic, Suture will ignore these attributes by default

Other attributes may or may not matter (for instance, other timestamp fields, or the id of the object), in those cases, you can instantiate the comparator yourself and tell it which attributes to exclude, like so:

Suture.verify :thing,
  :subject => Thing.new.method(:stuff),
  :comparator => Suture::Comparator.new(
    :active_record_excluded_attributes => [
      :id,
      :quality,
      :created_at,
      :updated_at
    ]
  )

If Thing#stuff returns an instance of an ActiveRecord model, the four attributes listed above will be ignored when comparing with recorded results.

In all of the above cases, :comparator can be set on both Suture.create and Suture.verify and typically ought to be symmetrical for most seams.

Examples

This repository contains these examples available for your perusal:

• A Rails app of the Gilded Rose kata

Troubleshooting

Some ideas if you can't get a particular verification to work or if you keep seeing false negatives:

• There may be a side effect in your code that you haven't found, extracted, replicated, or controlled for. Consider contributing to this milestone, which specifies a side-effect detector to be paired with Suture to make it easier to see when observable database, network, and in-memory changes are made during a Suture operation
• Consider writing a custom comparator with a relaxed conception of equivalence between the recorded and observed results
• If a recording was made in error, you can always delete it, either by dropping Suture's database (which is, by default, stored in db/suture.sqlite3) or by observing the ID of the recording from an error message and invoking Suture.delete!(42)

suture's Issues

Record a call

So, when the environment variable SUTURE_RECORD_CALLS is set to something truthy when setting up the suture, then every time :old gets invoked, record the call to the sqlite database

store schema table with major version

• create a table named schema_info and col major_version
• add a MAJOR_VERSION=1 constant to the adapter
• if they don't match, error out and recommend users downgrade/upgrade as needed

Implement Suture.create :call_both

When :call_both is set (intended for staging environments), the following is required:

• both :new and :old code paths must be set
• call_both and records_calls are mutually exclusive (ensure #15 includes a check for this)

And the following happens:

• the :new path is called and its result is held
• If #40's set to have a clean-between callable, call it
• the :old path is called and its result is held
• If #40's set to have a clean-after callable, call it
• the two results are compared using whatever comparator applies to the seam
• If the two results match, the new result is returned
• If they don't match and #39 is set to raise, then raise an error (either way log a warning)

Rails project

A sample Rails project that:

• leverages the built in comparator for simple AR result types
• includes a one-off comparator for tougher AR result comparison
• has a seam that takes some input (AR or params) and spits out an AR object

Log out activities

Suture should log out when it's called and records something:

[<time>] Suture recorded a call to on your :foo seam to the :old path with arguments [1, 2, 3] and result 6

Can be configured by Suture.config (see #25) or ENV vars, but unlike other options, can't be set on a per-suture/verify basis b/c the logger is going to be a singleton (just to prevent config & reference leaking of Plan and TestPlan models further, esp. when logging in the absence of one)

Settings:

Suture.config({
  :log_level => "INFO",
  :log_file => nil,
  :log_stdout => true
})

SUTURE_LOG_LEVEL, SUTURE_LOG_FILE, and SUTURE_LOG_STDOUT should also work.

Document rails env-specific settings

A lot of Suture's settings make more sense for one RACK_ENV/RAILS_ENV than another, and I imagine a lot of stuff (like logging or database_path) will be set differently in development.rb than test.rb or production.rb

There's nothing Suture needs to do to support this but it should probably be documented.

rake test -> rake unit w/ test as default

Rails project + production snapshot

Show a rails example that has a production postgres dump, sets up a one-off minitest rake task, can exploratory test the dump to generate a suture DB, and then run the verifications in a local test

Empahasize that you don't have to commit your production database--you only need it long enough to get and verify each path--that this stuff should be transient

Limit error messages printed by VerificationFailed

When verify fails on 4600 of 5000 calls, the user probably doesn't need to see all of them. We should have a setting like limit_failure_messages

Suture.verify :after_subject hook

Just like #40, some users will need a hook for cleaning up after side effects following each run.

Document configuration

Add to the README a list of all config options and:

1. What the defaults are
2. What can be set by ENV
3. What can be set by Suture.config / Suture.reset!
4. What can be set by Suture.create/verify

fail_fast configuration

• by default, fail fast and error immediately
• if fail_fast: is falsey, then aggregate failures and log them all at the end of the run.
• in addition to aggregating all the failure results, also print an overall progress
• bar_of_progress gem

API to drop Suture's database

Suture.drop_database or recreate_database or something may be appropriate, given that the only way to do it currently is to know where the file is and delete the file

Set SUTURE_ENV & raise/warn

This seems low priority and hard to get right.

Make SUTURE_ENV reflect RACK_ENV/RAILS_ENV convention of development, test, and production.

• Maps in our case to development => record mode, test => verify, and production => compare & retry modes.
• If user tries to record in a non-development env, warn them
• If a user tries to verify in a non-test env, warn.

Register a known side effect

It'd be neat to identify a method you associate with a side effect (like an http.post) and then wrap it. Still calling through but warning the user?

As for default registrations, I suspect vcr or webmock might have some prior art on this from an HTTP perspective? Would be good to have an API that could handle database/socket connections as well.

common test parent for unit tests

do this in global setup:

      # TODO: the following is necessary everywhere we log until we have a common test parent
      Suture.config(:log_stdout => false)
      Suture::Adapter::Log.reset!

API to delete a seam's recordings from the database

It'll probably be the case that a team working on multiple seams at a time will wind up with more than one seam at a time in Suture's database. Suture.delete(:seam_name) should be able to delete all the rows for a given seam

Verify: API to delete a row from calls

So when verify fails then the message should print instructions on how to delete the recorded call if it is known to be in error (or not worthwhile).

Example idea: Suture.delete(:my_name, 1234)

Logging configuration

• Should be able to log to a designated file and/or stdout.

We probably don't need multiple levels

Suture.create :raise_on_result_mismatch

When :run_both is true, this flag determines whether a Suture::Error::ResultMismatch is raised (default is true).

if it's false, presumably it's enough to log a warning

Verify a single call

• when verify fails, print the ID that failed as SUTURE_VERIFY_ONLY=1234
• if the env var is set, then only select that row+name combo
• if not found, raise error
• run just that example

Gilded Rose kata

Expand the safe test we currently have for gilded rose into a multi-step walkthrough of using Suture against a pure function.

Add global Suture.config method (and Suture.reset!)

Let folks override configuration for the life of the process. Any options set will be forwarded to Plan and TestPlan models and stick if they're valid.

Suture.config({:database_path => "some/file.db"})

Suture.create(…) # <- will use some/file.db

Suture.reset! #<- will revert to the default

Support permitted error types

Some code raises error intentionally for certain inputs and is the effective "return" value of a given call.

• Suture config should take a list of expected error types:

  Suture.create(:foo, {
    :old => ->{ raise MyErrorClass.new },
    :args => [],
    :expected_errors => [MyErrorClass, MyOtherError]
  })

• When recording the above, the "result" should be the error
• When verifying the above, if an error is rescued, check to see if it was expected and if it matches a permitted error type. Only mark the test result in error if it was not permitted.

Log out an error every time a suture raises

The Scalpel should log_error every time an error is raised from an invocation (I suppose by rescuing & re-raising).

• figure out how to test logs

Allow configuration of different sqlite location

Suture.config({
  database_location: "db/suture.sqlite3"
})

(Global Suture.config({}) method that merges input with a class-attr)

Add code coverage / code climate

implement changelog generator

https://github.com/skywinder/github-changelog-generator

Document how to use Coverage

Coverage itself is out of scope here, but a doc to show off how to use something like simplecov in conjunction with Suture would be useful in a guide.

Suture.create :call_old_on_error flag

When set, errors raised by :new will be rescued and :old will be called and its value returned instead.

Seems like this should work in conjunction with the staging-focused flags in #40 to clean up-between and after the two paths run.

Validate test plans

In particular, blow up early if subject isn't defined as that one is kinda required.

• subject is not defined or not callable

Verify: Implement call_limit

When Suture.verify is passed a call_limit => 5, then it will only run 5 examples and stop. If all 5 verifications pass, it's happy (i.e. otherwise everything is the same)

Get the example projects into a build

It'll slow travis way down to keep it in the main build -- possible to make a secondary build for them?

Override flag to disable the seam / revert to old path

If someone is working on master and wants to be able to push to production safely while the :new code path is still under development or otherwise not tested, let them hide it behind a flag that will override all other Suture options and guarantee to call the old code path and take no other action (the NoOp command)

What's a good name for this? SUTURE_DISABLE=true / :disable => true?

Verify: Implement random order

Suture.verify should have random as the default ordering.

• when selecting calls, do a pseudo-random ORDER BY (see this hacky solution for an idea)
• print the seed with any failures
• SUTURE_RANDOM_SEED= should override the seed

Record mode: Should we dup arguments?

There is a risk that legacy code will modify the arguments when we call. If we wait to record things until after invoking the subject method, then we could:

1. Immediately create the observations record and dump the args before calling the legacy code
2. Duplicate the args we persist from the ones we pass

Suture.create :after_old and :after_new hooks

When :runs_both is enabled, it may be necessary to clean up state between the old & new code paths to ensure both can run correctly, or—since the new path's result is going to be returned—the old path's side effects cleaned up after the fact

Since any harmful side effects by the old or new code paths will apply to each mode, it may as well be supported by all of our modes as simply after_old and after_new.

Hello world project

An absolutely minimal little project that incorporates Suture and shows its API configuration off at multiple lifecycle stages (how to do that at once is unclear to me right now)

Validate plans

When building the plan, there's a bunch of ways to become logically invalid, and we should fail fast and explain to folks.

Examples:

• if :old & :args are not set, then none of the surgeons are going to know what to do
• if :record_calls is true, but so is :retry or :try_both are also true, it'll not be clear what mode we're in

The issue is that the tool has multiple logical modes but it's not something as simple as could be resolved with an :env => "development", because that's ambiguous and/or would leave off other options.

Such is the burden of having a single public method that does 4 drastically different things.

Add uniq index on name+args

We want to make sure that name, args, and result have a unique constraint.

• If we try to insert and this is violated, then select that row and compare the results:
  • If the results match, it's just an extraneous record, so do nothing
  • if the results do not match, then raise, print the recorded & new result value and explain the mismatch and suggest the user figure it out (e.g. write a comparator or delete the old record with Suture.delete from #10)

Suture.verify method

• given a name, load all of the rows from the sqlite database
• for each row, invoke the callable with args & result
• compare each, raise a Suture::VerificationFailure error if it doesn't match

Add sensible defaults to active record objects

• By default, (recursively?) compare attributes, excluding id, created_at, and updated_at.
• initially implement with a devdep on AR, but replace it with a minimal fake once we're done to save on build time.

Reset logging config when resetting suture

Suture.reset! should give folks a second chance to config the logger if need be rather than living for the life of the process

Suture.create :on_{new,old,subject}_error handler

Add a custom on-error hook when the new code path fails in production, allowing folks to do something when the code path raises an (unexpected, see: #24 ) error.

on_new_error: -> (name, args, error) { PhonesHome.new.phone(name, args) } 

Its return value doesn't have any kind of impact on what happens next, Suture will propagate the error just as it would have

Document Value-ifying methods with side effects

Document this:

When a bit of legacy code has a side effect (e.g. gilded rose kata, which mutates an array of items on each no-arg call), one could write a new seam that analyzes this:

def gilded_rose_entry_point(items)
  gilded_rose = GildedRose.new(items)
  Suture.create(:gilded, {
    :old => ->(items) { gilded_rose.method(:update); items) },
    :args => [items]
  })
end

The above essentially "value-ifies" a method under test that takes no args & has no return value so it can be observed.

Crazy idea that won't work (?)

What if Suture supported this more explicitly with additional options recorded_args and calculated_result? The below example wouldn't really work on verify, because the state of items would need to be set up perfectly before verify began.

def gilded_rose_entry_point(items)
  gilded_rose = GildedRose.new(items)
  Suture.create(:gilded, {
    :old => gilded_rose.method(:update),
    :args => [],
    :recorded_args => [items],
    :calculated_result => -> { items }
  })
end

Create/Initialize a sqlite database

Look at default db location (db/suture.sqlite3), and if one does not exist:

• initialize a new db there
• run some pre-baked DDL to create a calls table (with name, recorded_at, env, args, result)

Verify: implement time_limit

Just like #7 but for time durations.

Support awesome_print

Error messages should use awesome_print if it's available and can be embedded into error messages strings easily/safely

Add comparator API

Idea here is to allow custom comparators to be configured on two dimensions:

• Globally via Suture.configure or locally via an option to Suture.create or Suture.verify
• At compare-time find the Plan's closest ancestor for which a custom comparator is defined

For instance, if I have this global config:

Suture.config({
  :comparators => {
    ActiveRecord::Base => ->(recorded, actual) {
      recorded.attributes.except(:id, :created_at, :updated_at) == 
        actual.attributes.except(:id, :created_at, :updated_at)
    }
  }
})

And this config inside a suture:

Suture.create(:load_user, {
  :comparators => {
    ActiveRecord::Base => ->(_,_){ raise "hell" },
    User => ->(recorded, actual){ recorded.id == actual.id }
  }
})

In the above case, the :load_user Suture would have the ActiveRecord::Base comparator overwritten with one that simply raises (for no reason here other than to illustrate that fact) and the User comparator which would "win", since (assuming it inherits from AR Base) it's the closest ancestor.

Create a new Suture

Be able to do this:

Suture.new(
  :old => MyThing.new,
  :args => [1,2,3]
)

Where old can be any call-able, this should work too:

Suture.new(
  :old => ->{|a,b,c|  MyThing.new.call(a, b, c)  },
  :args => [1,2,3]
)