nelyj/field_test

Bayesian A/B testing for Rails

Field Test

🍁 A/B testing for Rails

• Designed for web and email
• Comes with a handy dashboard
• Seamlessly handles the transition from anonymous visitor to logged in user

Uses Bayesian statistics to evaluate results so you don’t need to choose a sample size ahead of time.

Installation

Add this line to your application’s Gemfile:

gem "field_test"

Run:

rails g field_test:install

And mount the dashboard in your config/routes.rb:

mount FieldTest::Engine, at: "field_test"

Be sure to secure the dashboard in production.

Getting Started

Add an experiment to config/field_test.yml.

experiments:
  button_color:
    variants:
      - red
      - green
      - blue

Refer to it in views, controllers, and mailers.

button_color = field_test(:button_color)

When someone converts, record it with:

field_test_converted(:button_color)

When an experiment is over, specify a winner:

experiments:
  button_color:
    winner: green

All calls to field_test will now return the winner, and metrics will stop being recorded.

Features

You can specify a variant with query parameters to make testing easier

?field_test[button_color]=green

Assign a specific variant to a user with:

experiment = FieldTest::Experiment.find(:button_color)
experiment.variant(participant, variant: "green")

You can also change a user’s variant from the dashboard.

Config

By default, bots are returned the first variant and excluded from metrics. Change this with:

exclude:
  bots: false

Keep track of when experiments started and ended. Use any format Time.parse accepts. Variants assigned outside this window are not included in metrics.

experiments:
  button_color:
    started_at: Dec 1, 2016 8 am PST
    ended_at: Dec 8, 2016 2 pm PST

Add a friendlier name and description with:

experiments:
  button_color:
    name: Buttons!
    description: >
      Different button colors
      for the landing page.

By default, variants are given the same probability of being selected. Change this with:

experiments:
  button_color:
    variants:
      - red
      - blue
    weights:
      - 85
      - 15

If the dashboard gets slow, you can make it faster with:

cache: true

This will use the Rails cache to speed up winning probability calculations.

Funnels

You can set multiple goals for an experiment to track conversions at different parts of the funnel. First, run:

rails g field_test:events

And add to your config:

experiments:
  button_color:
    goals:
      - signed_up
      - ordered

Specify a goal during conversion with:

field_test_converted(:button_color, goal: "ordered")

The results for all goals will appear on the dashboard.

Analytics Platforms

You can also send experiment data to analytics platforms like Google Analytics, Mixpanel, and Ahoy. Use:

field_test_experiments

to get all experiments and variants for a participant and pass them as properties.

Security

Devise

authenticate :user, -> (user) { user.admin? } do
  mount FieldTest::Engine, at: "field_test"
end

Basic Authentication

Set the following variables in your environment or an initializer.

ENV["FIELD_TEST_USERNAME"] = "moonrise"
ENV["FIELD_TEST_PASSWORD"] = "kingdom"

Credits

A huge thanks to Evan Miller for deriving the Bayesian formulas.

TODO

• Code samples for analytics platforms

History

View the changelog

Contributing

Everyone is encouraged to help improve this project. Here are a few ways you can help:

• Report bugs
• Fix bugs and submit pull requests
• Write, clarify, or fix documentation
• Suggest or add new features