rawls238/react-experiments

React components for implementing UI experiments

react-experiments

This library is no longer actively maintained.

react-experiments is a set of React components for implementing UI experiments.

For additional context behind why we built this, check out the accompanying blog post

Installation

npm install react-experiments

Usage

If you are using React 0.14 the latest supported version is 4.1.0

react-experiments was built to work with PlanOut.js and most of its constructs are inspired by the structure of PlanOut.js. This library will work out of the box if you pass it an instantiated PlanOut Namespace or Experiment class, but if you want to use your own methods of assigning experiment parameters and logging exposure then you can extend the base experiment class and pass that as the experiment class prop.

Implementing a simple experiment

This library serves as a way to declaratively implement UI experiments that are defined via PlanOut. The standard usage of this library is as follows:

1. Define experiment via PlanOut script / API. The PlanOut parameters that you set should map to the props on which you want to run an experiment. Let's use the sample PlanOut.js experiment as an example, which is effectively:

signupText = uniformChoice(choices=['Signup', 'Join now'])

2. Wrap the component where you want to implement your UI experiment with the parametrize function provided by the library along with an instantiated experiment class and the specific parameter names that you want to parametrize the component with. As an example,

const Signup = parametrize(DummyExperiment, ['signupText'], React.createClass({
  render() {
    return (
      <div>
        {this.props.signupText}
      </div>
    );
  }
}));

Now you should be all set to run the sample experiment. The Signup component will render 'Sign up' or 'Join now' based on the randomized parameter assigned by PlanOut.js.

To put it all together,

exp = new DummyExperiment({ id: 'this_is_the_user_id'});

let Signup = parametrize(exp, ['signupText'], React.createClass({
  render() {
    return (
      <div>
        {this.props.signupText}
      </div>
    );
  }
}));

let Parent = React.createClass({
  render() {
    ...
    <Signup />
  }
});

Base Parametrize Component

The implementation of all the components provided by this library are wrappers around a base Parametrize component. The Parametrize component allows for parametrizing a given component with experiment parameters. The following are the props that the Parametrize component takes:

experiment: This is an instance of a PlanOut.js experiment/namespace class or the base Experiment class. [REQUIRED]

params: This is the list of experiment parameters that you want to use to parametrize the component. They should correspond to the parameter names defined in your PlanOut script / experiment definition. [REQUIRED]

[any arbitrary prop]: You can pass arbitrary props to the Parametrize component and they will be available via context.experimentProps in all descendants of the Parametrize component.

Higher-order Parametrization Components

There are two primary higher-order components to use for parametrization.

parametrize: The parametrize function takes an instantiated experiment class, either an experiment name or a list of params, and a React component. It takes the given component and sets the deterministically and randomly assigned experiment parameters of the experiment class as props.

parametrize(exp, ['signupText'], React.createClass({..}));

withExperimentParams: The withExperimentParams function is used in combination with the base Parametrize component. It is useful when running an experiment with nested components, but generally the parametrize function should be preferred.

const Parent = createReactClass({
  render() {
    return (
      <Parametrize experiment={exp} params={['signup_form_text', 'signup_nav_text']}>
        <SignupHeader />
        <SignupForm />
      </Parametrize>
    );
  }
});

const SignupHeader = withExperimentParams(createReactClass({
  render() {
    return (
      <div>
        {this.props.signup_nav_text}
      </div>
    );
  }
});

const SignupForm = withExperimentParams(createReactClass({
  render() {
    return (
      <div>
        {this.props.signup_form_text}
      </div>
    );
  }
});

Running A/B Variation experiments:

There are two common types of experimental parameters:

1. Parameters that correspond to parametrizations of existing variables and components. For instance, if one is running an experiment to test which shade of blue optimizes the click rate of the button, then the values to which your experiment parameters map would correspond to something such as the different hex codes for the different shades of blue.

2. "Branching" parameters where the parameter values correspond to different "variations" of the experiment. For instance, if one is testing two completely different user interfaces then it wouldn't make sense to parametrize every aspect that has changed, but rather to bin users into either 'Variation A' or 'Variation B'.

While the core component of this library focuses on the first type of parameter, it also includes some convenience components built around the Parametrize component for running "branching" experiments using the ABTest component.

<ABTest on='foo' experiment={TestNamespace}>
  <When value='foobar'>
    variation 1
  </When>
  <When value='bar'>
    variation 2
  </When>
  <When value='test'>
    variation 3
  </When>
  <Default>
    variation default
  </Default>
</ABTest>

The ABTest component above branches off the value of this.props.experiment.get(this.props.on);, TestNamespace.get('foo') in this case, and renders the When component where ABTest.props.experiment.get(ABTest.props.on) === ABTest.props.value. If it doesn't find a corresponding When component to render then the Default component will render. This component makes implementing an experiment using "branching" parameters easy.

The ABTest component takes the following as props:

experiment - an instantiated PlanOut namespace/experiment class or a custom Experiment class. [REQUIRED]

on - the parameter name to "branch" off [REQUIRED]

Customized Experiment Components

If you want to create your own experiment component you can extend the base Parametrize component.

Logging

react-experiments deals with logging experiment exposure. All the components provided by this library trigger an exposure log when the component is mounted.

Development

This project is written using ES6 and all build steps / transpilation are done by webpack. Be sure to run npm install to install the necessary development dependencies. In order to develop and test locally, it is necessary to simply run the build.sh shell script which will take care of transpiling to ES5 and running tests.

To test API changes locally, open the examples/index.html file locally after building with your most recent changes. The index.html file contains a simple example of using this library paired with the PlanOut.js sample experiment.

Please be sure to add tests to this project when making changes. This project uses Jest for running tests. Tests can be run either by building the project using build.sh or by using npm test.

react-experiments's Issues

Examples for tracking

Would love to see how each variation chosen by planout can be instrumented with tracking tools such as GA or Amplitude.

A few ideas / thoughts

This is awesome! React and PlanOut always seemed like they could work nicely together, in that PlanOut randomizes parameter values and React components are parametric (via props).

A few ideas / comments:

(1) Experiment component: A React component's render() method should not modify state, but renderExposedVariation(), which gets called by render affects state. It also seems less than ideal to have to re-evaluate all of the logic in renderExposedVariation() every time the component needs to be re-rendered. Why not just save the component you want to be rendered to state in getInitialState() or componentWillMount()?

One thing to watch out for is that you want to make sure that your logging does not depend on your components successfully rendering -- otherwise you might not catch cases where your experiment is causing bugs, and you'll get non-equivalent users in your different experimental conditions.

(2) Parameters as props: part of the idea behind PlanOut is to get people thinking about experiments in terms of parameterizations of user experiences rather than variants. Such a notion seems compatible with the way one generally writes React code, in that components' behaviors or features depend on props.

You might do this generically by saying something like:

<Experiment experimentClass={foo}>
  ...
  <Parameterize propName1='param1', propName2='param2'>
    <MyComponent/>
  </Parameterize>
  ...
  <Parameterize propName3='param3'>
    ...
  </Parameterize>
  ...
</Experiment>

Where, for example, MyComponent is a component that includes two props, propName1 and propName2, which are populated with param1 and param2 from the experiment.

(3) Variations: I would suggest not marrying experiments to a single parameter, and instead having a <Variations> component that performs a function similar to what the existing Experiment component does, e.g.,

<Experiment experimentClass={foo}>
  <Variations variationParam='show_text'>
    <Variant name='experimental'>
      Example A
    </Variant>
    <Variant name='control'>
      Example B
    </Variant>
    <DefaultVariant>
      Things are broken.
    </DefaultVariant>
  </Variations>
</Experiment>

(4) Logging: If I understand correctly, you are only logging when there is a matching variant. This seems problematic...

(i) In the case of the logging example in the README, it's not clear what the treatment/control contrast is. Under the given scheme, it seems like you would likely only be logging / measuring effects on users who are not necessarily equivalent because those who hit different endpoints could be very different from one another.

(ii) Logging conditional on matching a known variant is also problematic when the components are causally downstream of one another. Let's imagine that you have some encouragement on page X that offered a discount (call this D=1), and we only expressed a variant to be shown when D=1 (so that only D is logged only when it equals 1), and then on page Y, the purchase form, we either shown the original price (D=0) or the discounted price (D=1). Visitors with D=0 are folks who would have visited Y regardless of the discount, while those with D=1 include marginal users who would not have otherwise visited (and may therefore be less likely to convert or spend as much). This means that even if you were to log all users who visit Y, those with D=0 and D=1 represent non-random populations. This is only an issue because you didn't exposure log all users who hit X because we were only logging users who had a matching variant at the time.

(5) Namespace component: Seems interesting, but doesn't seem practical from a code maintainability / readability perspective. I also wonder how this would play out as you add and remove experiments.

(6) Have you done much with Flux? I wonder how transportable this is to that kind of model.

Overall really neat stuff, I am looking forward to seeing how things progress with this!

More Examples

Would be great to have a few more concrete examples, especially examples with namespaces.

Would also be cool to have a simple interactive app that generates the jsx for a particular experiment definition

cc @mattrheault

extra <span> after <Parametrize>

so the case is everytime i put injectedExperiment the component are wrapped with extra , i check it with react-dev-tools
The issue as in example

<Parametrize props= props=>
 <span> #extraSPAN issue
  <imageComponent />
 </span>
</Parametrize>

and this issue make the component styling a mess which is really annoying. my workaround for this are passing it from another component, so the experimentParams are passed from parent component.

Triggering conversions?

If I have a revenue event at the end of a flow and multiple experiments within the test, how can I get an all-inclusive list of the best performing experiments?

React 16 Support

tracking for single page app?

Is there anything in react-experiments that would help us with page tracking on a single page app? We are desperately needing asynchronous page tracking.

error/warnings loading example

cloned, npm install, ran build.sh, then opened examples/index.html--it may be something trivial since I'm pretty inexperienced, but hoping someone has dealt w/ this before.

React v15

verify compatibility / upgrade if needed

Setup instructions

Hi @rawls238, I was interested in tinkering around your module this weekend, but after checking out the repository, I am not quite sure how to get started with development. Perhaps you could add a contributing / development section to the README that explains the flow for getting started, and maybe includes a sample app?

Setup travis-ci

Set up https://travis-ci.org/

The build command should just be sh build.sh

cc @mattrheault

Create an example with logging

One of the things that is complicated about UI-side experimentation is that it requires a special library / system for sending exposure logs to the server. It would be nice if there were an example of how to do this.