dollarshaveclub/study

A simple, progressive, client/server AB testing library 📚

A progressive, client/server AB testing library.

Study is an AB testing library designed to be clear, minimal, and flexible. It works in both the server and browser with the use of driver-based persistence layers.

You can download the compiled javascript directly here

• Features
• Installing
• Usage
• API
• Guide/FAQ
• License

---

Features

• Powerful, clear API
• Many variations. ABCD testing
• Intelligent weighted bucketing
• Browser & Server support
• Storage Drivers: localStorage, cookies, memory, or build your own
• Well documented, tested, and proven in high production environments
• Lightweight, weighing in at ~ 3.8kb.
• Not tested on animals

Installing

# Via NPM
npm i studyjs --save

# Via Bower
bower i studyjs --save

# Via Yarn
yarn add studyjs

Developing

npm install # Install dependencies
npm build # Build the babel'd version
npm lint # Run linting
npm test # Run tests

Usage

<script src="study.js"></script>
<script>

  // Set up our test API
  const test = new Study({
    store: Study.stores.local
  });

  // Define a test
  test.define({
    name: 'new-homepage',
    buckets: {
      control: { weight: 0.6 },
      versionA: { weight: 0.2 },
      versionB: { weight: 0.2 },
    }
  });

  // Bucket the user
  test.assign();

  // Fetch assignments at a later point
  const info = test.assignments();
</script>

API

Study(config)

const study = new Study({
  debug: true,
  store: Study.stores.local
});

This creates a new test API used to defined tests, assign buckets, and retrieve information.

Returns: Object

Name	Type	Description	Default
debug	Boolean	Set to true to enable logging of additional information	false
store	Object	An object with get/set properties that will accept information to help persist and retrieve tests	Study.stores.local

---

study.define(testData)

// Create your test API
const study = new Study();

// Define a test
study.define({
  name: 'MyTestName',
  buckets: {
    variantA: { weight: 0.5 },
    variantB: { weight: 0.5 },
  },
});

This function defines the tests to be assigned to used during bucket assignment. This function accepts an object with two keys, name and buckets. Alternatively, you may pass an array of similar objects to define multiple tests at once.

The name value is the name of your test. The keys within bucket are your bucket names. Each bucket value is an object containing an object with an optional key weight that defaults to 1.

The percent chance a bucket is chosen for any given user is determined by the buckets weight divided by the total amount of all weights provided for an individual test. If you have three buckets with a weight of 2, 2/6 == 0.33 which means each bucket has a weight of 33%. There is no max for the total weights allowed.

Returns: null

Name	Type	Description	Default
data	Object/Array	An object/array of objects containing test and bucket information	null

---

study.assign(testName, bucketName)

const study = new Study();
study.define({
  name: 'new-homepage',
  buckets: {
    variantA: { weight: 0.5 },
    variantB: { weight: 0.5 },
  }
});

// Assign buckets from all tests to the user...
study.assign();

// or assign bucket from the specified test...
study.assign('new-homepage');

// or specify the bucket from the specified test...
study.assign('new-homepage', 'variantB');

// or remove the bucketing assignment from the specified test.
study.assign('new-homepage', null);

Calling the assign method will assign a bucket for the provided tests to a user and persist them to the store. If a user has already been bucketed, they will not be rebucketed unless a bucketName is explicitly provided.

If no arguments are provided, all tests will have a bucket assigned to the user. If the first argument provided is a test name, it will attempt to assign a bucket for that test to a user. If a bucketValue is provided, it will set that user to the specified bucket. If the bucketValue is null, it will remove that users assignment to the bucket.

Returns: null

Name	Type	Description	Default
testName (optional)	String	The name of the test to assign a bucket to	null
bucketName (optional)	String	The name of the bucket to assign to a user	null

---

study.definitions()

const study = new Study();
study.define({
  name: 'new-homepage',
  buckets: {
    variantA: { weight: 0.5 },
    variantB: { weight: 0.5 },
  }
});

// Retrieve all of the provided tests
const tests = study.definitions();

This provides the user with all of the tests available.

The returned information will be an array if multiple tests were defined, otherwise, it will be an object of the single test defined. The object will mirror exactly what was provided in the define method.

Returns: Object|Array

---

study.assignments()

const study = new Study();
study.define({
  name: 'new-homepage',
  buckets: {
    variantA: { weight: 1 },
  }
});

// Capture assignments
study.assign();

// Retrieve all of the bucket assignments for the user
const buckets = study.assignments();
assert.strictEqual(buckets['new-homepage'], 'variantA');

This provides the user with all of the bucket assignments for the current user.

The returned information will be an object whose keys will be test names and values will be the current bucket assigned to the user.

// Example return
{
  'new-homepage': 'variantA',
  'some-test': 'some-bucket',
}

Returns: Object|Array

---

study.extendAssignments

Extending assignments can be a useful way to augment your Study implementation with third party software.

const study = new Study();

// Create a function that will modify assignments before you call `assignments`
study.extendAssignments =
  (assignments) => Object.assign(assignments, { foo: 'bar' })

// Retrieve all of the bucket assignments for the user
const buckets = study.assignments();
assert.strictEqual(buckets['foo'], 'bar');

A more practical example could be to implement with a third party AB testing platform like Optimizely (This uses pseudo code for brevity)

study.extendAssignments = (assignments) => {
  if (window.optimizely)
    for (const experiment in optimizely.experiments())
      assignments[experiment.name] = experiment.bucket

  return assignments
}

Returns: Object

---

Guide/FAQ

CSS Driven Tests

Tests logic may be potentially powered on solely CSS. Upon calling assign, if the script is running in the browser, a class per test will be added to the body tag with the test name and bucket in BEM syntax.

<body class="new-homepage--variantA"> <!-- Could be new-homepage--variantB -->

.new-homepage--variantA {
  /* Write custom styles for the new homepage test */
}

Storing metadata associated with tests

Each bucket provided may have additional metadata associated with it, and may have its value retrieved by retrieving the assignments and definitions.

const study = new Study();
study.define({
  name: 'new-homepage',
  buckets: {
    variantA: { weight: 1, foo: 'bar' },
  }
});

study.assign();

const defs = study.definitions();
const buckets = study.assignments();
const bucket = buckets['new-homepage'];
const bar = defs.buckets[bucket].foo; // "bar"

License

MIT Licensing

Copyright (c) 2015 - 2018 Dollar Shave Club

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

study's Issues

v4

• store each bucket separately
• server-side support
• create a server-side build (i.e. just babel-ifying)

Math error

const rand = (min, max) => Math.random() * ((max - min) + min);

should be

const rand = (min, max) => Math.random() * (max - min) + min;

if the bucket weight is set to 0 and that bucket was persisted, it should unset that bucket

ex. if a user was bucketed to challenger, then later we set the weight of challenger to 0, the user should be unbucketed from challenger the next time study is called

Mixed Content

Started seeing a Mixed Content error.

Can the protocol change to secure?
Instead of this: http://i.imgur.com/lnDx7VT.jpg
Use this: https://i.imgur.com/lnDx7VT.jpg

Analytics?

Hi there

Noob Question: is there a way to have the data from the A/B tests reported inside of Google Analytics?

Thanks!

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on all branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please delete the greenkeeper/initial branch in this repository, and then remove and re-add this repository to the Greenkeeper App’s white list on Github. You'll find this list on your repo or organization’s settings page, under Installed GitHub Apps.

Gh-page ??? 🤤

Cleanup assignments once experiment no longer exists

Requested Update

studyjs is behaving correctly when a user has an experiment and group defined but the experiment no longer exists. However the value remains in the store.
I'd suggest that once experiments are no longer defined when calling .assignments() that the group assignment is removed from the store.

File Size

Original Study size was 4.4k (minified). With the inclusion of Webpack, the dist files have ballooned to ~ 13k (minified). Is webpack necessary?

Bower badge version not working

This usually means there is a bower publishing issue.

refactor

• separate utilities to separate files
  • should be fine to use if we do a UMD build
• remove on body ready logic
  • this should be handled by the application
  • unnecessary if we load the script tags in the body
  • alternatively, we can just add classes to the html element
• remove response
• remove stores, default to memory
  • easier to test when we don't have to deal with localStorage
  • NOTE: NO DEFAULT
• tests – run on JSDOM so it's easier
• consider removing bower and only support npm

[packages] post Greenkeeper, some packages still need ⬆️

Requested Update

Post adding Greenkeeper some packages still need to be updated

Why Is This Update Needed?

In order for newer versions of Webpack, Babel-loader, and other such goodnesses to work, @babel/* packages need to be used.

Are There Examples Of This Requested Update Elsewhere?

Yep, see here and here from DSC's Postmate repository.

Read about references issues here. Provide paragraph text responses to each header.

assign() bug

Hello,

When calling study.assign(testName) the control always goes into the store. No matter what weight you specify or how many times you reload.

I took the following steps:
-mkdir test
-npm init
-npm i studyjs --save

You can reproduce this by using the code in the readme.

<!DOCTYPE html>
<html lang="en">
<head>
	<meta charset="UTF-8">
	<title>Document</title>
</head>
<body>
	<script src="node_modules/studyjs/build/study.js"></script>
	<script>
		// Set up our test API
		  const test = new Study({
		    store: Study.stores.local
		  });

		  // Define a test
		  test.define({
		    name: 'new-homepage',
		    buckets: {
		      control: { weight: 0.6 },
		      versionA: { weight: 0.2 },
		      versionB: { weight: 0.2 },
		    }
		  });

		  // Bucket the user
		  test.assign('new-homepage');
	</script>
</body>
</html>

console.log(localStorage.getItem('ab-tests')) // returns "{"new-homepage":"control"}"

study.assign(testName) returns control 100% of the time

Featured Link

Featured link goes to the 404. 🔧