facebookarchive/planout

PlanOut is a library and interpreter for designing online experiments.

PlanOut

PlanOut is a multi-platform framework and programming language for online field experimentation. PlanOut was created to make it easy to run and iterate on sophisticated experiments, while satisfying the constraints of deployed Internet services with many users.

Developers integrate PlanOut by defining experiments that detail how units (e.g., users, cookie IDs) should get mapped onto conditions. For example, to create a 2x2 experiment randomizing both the color and the text on a button, you create a class like this in Python:

class MyExperiment(SimpleExperiment):
  def assign(self, params, userid):
    params.button_color = UniformChoice(choices=['#ff0000', '#00ff00'], unit=userid)
    params.button_text = UniformChoice(choices=['I voted', 'I am a voter'], unit=userid)

Then, in the application code, you query the Experiment object to find out what values the current user should be mapped onto:

my_exp = MyExperiment(userid=101)
color = my_exp.get('button_color')
text = my_exp.get('button_text')

PlanOut takes care of randomizing each userid into the right bucket. It does so by hashing the input, so each userid will always map onto the same values for that experiment.

What does the PlanOut distribution include?

The reference implementation for PlanOut is written in Python. It includes:

• Extensible classes for defining experiments. These classes make it easy to implement reliable, deterministic random assignment procedures, and automatically log key data.
• A basic implementation of namespaces, which can be used to manage multiple mutually exclusive experiments.
• The PlanOut interpreter, which executes serialized code generated by the PlanOut domain specific language.
• An interactive Web-based editor and compiler for developing and testing PlanOut-language scripts.

Other production-ready versions of PlanOut are available for Java, JavaScript, and PHP, and can found in the java/, js/, and php/ directories, respectively.

The alpha/ directory contains implementations of PlanOut to other languages that are currently under development, including Go, Julia, and Ruby.

Who is PlanOut for?

PlanOut designed for researchers, students, and small businesses wanting to run experiments. It is built to be extensible, so that it may be adapted for use with large production environments. The implementation here mirrors many of the key components of Facebook's Hack-based implementation of PlanOut which is used to conduct experiments with hundreds of millions of users.

Full Example

To create a basic PlanOut experiment in Python, you subclass SimpleExperiment object, and implement an assignment method. You can use PlanOut's random assignment operators by setting e.varname, where params is the first argument passed to the assign() method, and varname is the name of the variable you are setting.

from planout.experiment import SimpleExperiment
from planout.ops.random import *

class FirstExperiment(SimpleExperiment):
  def assign(self, params, userid):
    params.button_color = UniformChoice(choices=['#ff0000', '#00ff00'], unit=userid)
    params.button_text = WeightedChoice(
        choices=['Join now!', 'Sign up.'],
        weights=[0.3, 0.7], unit=userid)

my_exp = FirstExperiment(userid=12)
# parameters may be accessed via the . operator
print my_exp.get('button_text'), my_exp.get('button_color')

# experiment objects include all input data
for i in xrange(6):
  print FirstExperiment(userid=i)

which outputs:

Join now! #ff0000
{'inputs': {'userid': 0}, 'checksum': '22c13b16', 'salt': 'FirstExperiment', 'name': 'FirstExperiment', 'params': {'button_color': '#ff0000', 'button_text': 'Sign up.'}}
{'inputs': {'userid': 1}, 'checksum': '22c13b16', 'salt': 'FirstExperiment', 'name': 'FirstExperiment', 'params': {'button_color': '#ff0000', 'button_text': 'Sign up.'}}
{'inputs': {'userid': 2}, 'checksum': '22c13b16', 'salt': 'FirstExperiment', 'name': 'FirstExperiment', 'params': {'button_color': '#00ff00', 'button_text': 'Sign up.'}}
{'inputs': {'userid': 3}, 'checksum': '22c13b16', 'salt': 'FirstExperiment', 'name': 'FirstExperiment', 'params': {'button_color': '#ff0000', 'button_text': 'Sign up.'}}
{'inputs': {'userid': 4}, 'checksum': '22c13b16', 'salt': 'FirstExperiment', 'name': 'FirstExperiment', 'params': {'button_color': '#00ff00', 'button_text': 'Join now!'}}
{'inputs': {'userid': 5}, 'checksum': '22c13b16', 'salt': 'FirstExperiment', 'name': 'FirstExperiment', 'params': {'button_color': '#00ff00', 'button_text': 'Sign up.'}}

The SimpleExperiment class will automatically concatenate the name of the experiment, FirstExperiment, the variable name, and the input data (userid) and hash that string to perform the random assignment. Parameter assignments and inputs are automatically logged into a file called firstexperiment.log'.

Installation

You can immediately install the reference implementation of PlanOut for Python using pip with:

pip install planout

See the java/, php/, js/, alpha/golang, alpha/ruby, alpha/julia directories for instructions on installing PlanOut for other languages.

Learn more

Learn more about PlanOut visiting the PlanOut website or by reading the PlanOut paper. You can cite PlanOut as "Designing and Deploying Online Field Experiments". Eytan Bakshy, Dean Eckles, Michael S. Bernstein. Proceedings of the 23rd ACM conference on the World Wide Web. April 7–11, 2014, Seoul, Korea, or by copying and pasting the bibtex below:

@inproceedings{bakshy2014www,
	Author = {Bakshy, E. and Eckles, D. and Bernstein, M.S.},
	Booktitle = {Proceedings of the 23rd ACM conference on the World Wide Web},
	Organization = {ACM},
	Title = {Designing and Deploying Online Field Experiments},
	Year = {2014}
}

planout's Issues

Make a release for 0.6.x?

I noticed some version bumps in git master, but the PyPI version is still at 0.5 which was released in 2014. Any chance a new release would be released to bring Python 3 support and more?

High fives guys!

I'm totes excited about using this.

Suppress some intermediate variables from being returned in the parameter map?

When playing with the online PlanOut editor, I noticed that every variable ever declared in a PlanOut script gets returned in the resulting parameter map. However, judging from the examples in the language reference, it seems that some users may not want or do not care about having some of the declared variables as parameters because they might have been intermediate values used to calculate the final desired parameters.

Again going back to my whitelist use case, suppose that now we want to whitelist a certain list of entities into a particular treatment, which is declared in a variable for clarity and reusability, and this list grows very large. If this very large list now has to be sent back to the client every time a treatment is requested for this experiment definition, it could potentially cause a lot of network overhead.

One could perhaps argue that this is not a realistic use case, and that might very well be true (my understanding of the framework could still use some improvement). However, I am envisioning a scenario where all the experiment logic is centralized in a single service as part of a service-oriented architecture (see #111), and clients are given a simple interface to request experiment treatments from this central service. We are now faced with the following tradeoff:

• The server can by default return the entire parameter result (in JSON, perhaps) back to the client for the client to parse out the values it needs, in which case for large intermediate values there is the problem of network overhead,
• The server can expose some sort of interface (probably an argument to the API call) for the client to specify which parameters it cares about and do the filtering server-side, but in this case the client will have to be explicit about which parameters it wants.

I think the second path is the lesser of evils and is probably workable (would need to look into it in more detail later), but perhaps there are other situations as well where intermediate values could clutter the final parameters return value.

Can't replicate and operate on hashes in restricted languages -- Lua, JavaScript

Lua and Javascript don't have a dedicated integer type, and use "Number" types instead.

These rely on a 64 bit floating point, which can only use integers with precision up to 2^53. The hash hex string slices are currently taking 15 characters, for 2^60.

I'd suggest changing the slice of the hex from 15 to 12 characters, giving us 48 bits and a length still divisible by 16, fitting into numbers < 2^48.

How to update namespace experiments on the fly

I read document but I don't see how to update namespace expriments( number of segment in each experiment). From my understand I have to put class in self.add_experiment(exp_name, exp_class, n_segment) so how to add_experiment by SimpleInterpretedExperiment

Thank you.

How to do downstream analysis of PlanOut experiments? Any tools available?

Checking both the implementation of PlanOut and the PlanOut paper, it's NOT clear how to do downstream analysis of PlanOut experiments programmatically / automatically? Any tools out there available?

Can anyone please shed some light on this? Thanks very much!

Serialized format for Namespaces?

Is there an equivalent to the Planout experiment serialized JSON for namespaces?

Planout4j has something close.

I'm interested in defining and manage namespaces somewhere, and having an interpreter take in some JSON and construct an instance of a namespace (the same way you would create an Experiment instance from JSON).

note on logging issue

During the Pydata talk today, some users in the audience noticed that, on ipython notebooks, the logging can echo to the notebook, in addition to logging to a file. This is most likely a result of inheriting a parent log, depending on the user's configuration. In order to remove this, the following code can be run:

(assuming we have an experiment defined as 'e')

l = e.logger.values()[0]
l.parent.removeHandler(l.parent.handlers[0])

This grabs the logger from e, then removes the stream handler from the parent logger (which has been inherited). I figured this would be useful to you in future demos, since you mentioned noticing this issue for a small subset of users in the past.

Once this parent logger has been removed, it seems to be gone for all future experiments that are created.

PlanOut Compiler Demo broken on Firefox 41.0.2

OS: Mac, Yosemite 10.10.4

It works correctly on Safari.

https://facebook.github.io/planout/demo/planout-compiler.html

Python 3 iteritems call on dictionary throws error

Hello, I'm not quite sure if I have something set up wrong or this bug went unnoticed.

I get the following error when I try to remove an experiment from a namespace:

File "planout/namespace.py", line 152, in remove_experiment
    [s for s, n in self.segment_allocations.iteritems() if n == name]
AttributeError: 'dict' object has no attribute 'iteritems'

setup_experiments look like this:

def setup_experiments(self):
  self.add_experiment('gif', GifExperiment, 50)
  self.add_experiment('noconvo', NoConvoExperiment, 50)
  self.remove_experiment('gif')

Am I doing something wrong or should this be fixed by:

• using .items(), or
• using six.iteritems()

Namespace autoExposureLogging is broken

We ran into this today in the JS implementation...I looked at the Python implementation and I think it suffers the same issue...

Here are the steps to reproduce:

1. Have auto exposure logging enabled on all the experiments in a namespace
2. Get an experiment parameter for experiment A
3. For a user that is enrolled in experiment B, it simply checks that _experiment is set to something and tries to get the param from _experiment (in this case, experiment B).
4. When it tries to get the experiment parameter it automatically logs exposure for experiment B, even if the requested parameter is null or doesn't exist and the intention was to get the experiment parameter from experiment A, exposure was logged on experiment B.

Exposure shouldn't be logged in this case on experiment B, but given the existing implementation (at least in the JS + Python implementations) it is logged. Let me know if you agree that this is a bug or if perhaps something is wrong with how we are using namespaces.

@eytan

How to update experiment configuration without code deploy?

To whom it may concern,

I found this project while investigating A/B testing frameworks, and while it seems to provide a great deal of the functionality which I am looking for, there is one big design question that seems to remain unanswered upon looking at the documentation and skimming through part of the implementation (actually two, but they are related):

• It doesn't seem to be possible to be able to update the configuration of an experiment without modifying the code. For example, suppose you want to expose 10% of the population to some treatment, and then you want to increase that percentage to 25% for example. This seems to not be doable without updating the actual code. While this could be made to work reasonably well in environments that have continuous deployment or something close to it, perhaps in some organizations the deployment architecture / policies may not be able to easily accommodate this, slowing down the iteration process on experiments (and having to basically deploy a "hotfix" if some experiment needs to be rolled back).
  • As an added bonus, I would also like to have all of the original 10% that was seeing the treatment before to be part of the 25% in the updated experiment. This probably warrants a top-level bullet point in itself actually, though there might be ways to accomplish this just using the PlanOut language.
• Related to the above is the fact that Planout doesn't seem to be storing anything in any sort of data store (such as a database). Is this intentional, and if so, why? For example, the above problem regarding modifying experiment parameters (a form of versioning) could be solved if the experiment configuration were being stored somewhere outside of the actual application code. As of now, it seems that PlanOut is able to achieve a lot of the standard experimentation functionality (whitelisting via overrides, targeting, and namespacing) all without any sort of persistent storage, which I did not think was possible!
  • One consequence of not using any central datastore is that all salts must be stored locally on every instance that is running the code rather than consulting a central master server for getting experiment information (as well as the actual computation of assignment treatments). I am surprised that this does not cause any problems in a massive service-oriented architecture environment where all the experiment-related functionality is decentralized throughout all the servers?

Wondering how Facebook or other users get around these apparent limitations (maybe some more that I will think of later), but apart from these much of the rest of the system looks fairly clean yet sophisticated!

Ruby fix for WeightedChoice did not make it into the Gem

>$ gem install planout

Installing PlanOut from the gem seems to be missing this commit: d361582

cc: @eytan @mohnish

Create a ProductionExperiment class

The SimpleExperiment class is designed to make it easy to get up and running with PlanOut but it's not what people should be using for production. For example, experiments should be able to assert what variables they set, as to ensure the sanity of experiments when calls to the experiment request parameters that are not defined. This is done at Facebook via a getParamNames() method which reads from metadata that gets saved alongside experiments. While this might be overkill for many simple applications (and so we may not want to make it part of the SimpleExperiment class), it should be strongly encouraged for production scenarios.

This task is for creating a DemoProductionExperiment that includes this additional functionality.

Serialized planout code and Namespaces

Hello,

Sorry if this is a stupid question but how can I generate JSON serialized planout code when experiment namespaces are involved?

For instance, how can I represent demos/demo_namespace.py in JSON?

class DefaultButtonExperiment(DefaultExperiment):
  def get_default_params(self):
    return {'banner_text': 'Generic greetings!'}

class ButtonNamespace(SimpleNamespace):
  def setup(self):
    self.name = 'my_demo'
    self.primary_unit = 'userid'
    self.num_segments = 100 
    self.default_experiment_class = DefaultButtonExperiment

  def setup_experiments(self):
    self.add_experiment('first version phase 1', V1, 10) 
    self.add_experiment('first version phase 2', V1, 30) 
    self.add_experiment('second version phase 1', V2, 40) 
    self.remove_experiment('second version phase 1') 
    self.add_experiment('third version phase 1', V3, 30) 

Unit test pretty printing

Planout compiler package.json

Hello,

I'm building a custom compiler interface to integrate the experiments creation on a database in order to be able to distribute them across a number of applications.

Can we put the planout compiler on npm so anybody can npm i planout-compiler --save and require it via node's require?

PEP8 compliance

Is there a good reason to use 2 spaces instead of 4? This will break all the awesome tools that us Python devs use to check for PEP8 compliance. PEP8 should be followed unless you have a very good reason not to, and it seem like other Facebook Python libraries use the correct convension.

enable repository for travis

#101 purportedly added support for travis testing on the reference Python implementation but it's still necessary to turn the repository on for testing on travis-ci.org in order for the tests to actually run

How should namespace segments be used?

Namespaces are awesome but confusing

I realize that using namespaces allows me to run multiple experiments while ensuring that users are assigned to experiments in a mutually exclusive fashion. I love it and need it for that feature. The following behaviors that I observed confused me.

Should num_segments represent an estimation of the total population of users?

The primary unit in the experiment is the user ID. We have about 1,000,000 monthly unique visitors to our website where we are running this experiment. So I thought I should set the num_segments to something in the 100,000s so that I'll have sample size large enough to give us high confidence in measuring the impact of the experiment. When I set the num_segments to something in that range, it seems that the Namespace really slows down. Should that be the case?

Is the default_experiment only used when the total of segments allocated to experiments is less than num_segments?

Is the allocation of the segments to the running experiments simply a method of determining the distribution of users to experiments?

• For instance, if I had num_segments = 100, and assigned experiment 1 to 40 segments and experiment 2 to 60 segments, would I see roughly a 40% allocation of users to experiment 1?
• With num_segments set to 100, and 20 allocated to experiment 1 and 20 allocated to experiment 2, would I see 60% of users assigned to the default experiment, if I defined one?

Open sourcing Planout interpreter in Golang

https://github.com/URXtech/planout-go-interpreter

Hey guys, we (at URX) have implemented a Planout interpreter in Golang. Check it out! Any feedback/suggestion is greatly appreciated.

Multi-armed bandit?

Is it possible to use a multi-armed bandit algorithm? (Like http://stevehanov.ca/blog/index.php?id=132)

Alpha ruby implementation

Hey,

I understand that the ruby implementation is a literal port of the reference implementation. I guess there's no concrete plan or roadmap to move that out of the alpha stage?
Still maybe somebody can shed some light on what's missing for it to be moved out?

Just by reading through it I would expect better test coverage, docs and some tweaks here and there.

Conda

I don't know if this is under your control, but the conda/binstar for this package is way behind pip.

Thoughts on allowing dot notation in get ops

Hi all

I was integrating planout today with my application, and it occurred to me that it would be easier for integration to allow for dot operators in the grammar.

For example

recommend_page = bernoulliTrial(p=0.99, unit=[userContext.userid, request.pageid]);

In the salting side, I think the dots can be considered as strings just like before ?

"determnisitcally" is not a word

In index.md in the gh-pages branch.

Log analysis

Guys hi,

I did not find planout related groups or chats so asking a question here.
Is there any open source tools/frameworks you can recommend for analysis of PlanOut experiments logs?
Or you have to write your own?

I've found slides on the topic by Sean Taylor. But can't find presentation(video) itself.

Idiosyncratic compilation of array indexing

Currently the PlanOut compiler only supports variables to be on the left hand side of array indexing expressions. e.g.,

y = x[0];

works correctly, but

y = x[0][1];

doesn't work:

/Users/ebakshy/planout_fb/compiler/planout.js:193
        throw new Error(str);
              ^
Error: Parse error on line 1:
y=x[0][1];
------^
Expecting 'IDENTIFIER', 'END_STATEMENT', ']', ')', '}', '%', '/', '>', '<', 'EQUALS', 'NEQ', 'LTE', 'GTE', '+', '-', '*', 'OR', 'AND', ',', 'SWITCH', 'IF', 'ELSE', 'THEN', got '['
    at Object.parseError (/Users/ebakshy/planout_fb/compiler/planout.js:193:15)
    at Object.parse (/Users/ebakshy/planout_fb/compiler/planout.js:251:22)
    at Object.commonjsMain [as main] (/Users/ebakshy/planout_fb/compiler/planout.js:716:27)
    at Object.<anonymous> (/Users/ebakshy/planout_fb/compiler/planout.js:719:11)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)

Broken link on facebook.github.io/planout

If you click on PlanOut Editor it redirects to a relative URL and goes to the facebook.github.io homepage. Should instead go link to http://planout-editor.herokuapp.com/ so that it redirects properly

How does JSON literals work

Hi all - I am currently writing a PEG based parser for planout in python to compliment the one currently written in javascript.

I have everything working as expected, but I dont understand the JSON literal. Is there any example of this ?

Discussion of PO java port

Moving the conversation between @eytan and @eric239 about open-sourcing PO java port here for better tracking.

(Minor) Not obvious how to check for list membership

After looking at the language reference and the compiler (https://github.com/facebook/planout/blob/master/planout-editor/js/utils/planout_compiler.js), the way to check for existence of something in a list is not well documented.

One use case I can think of is in case I want to implement whitelists and/or custom targeting. This is a completely fabricated example (though intended to mimic real world use cases), but let's say I want to show a treatment only to users in certain countries, and I am able to pass the country in as an argument. Here is a workaround that worked when quickly playing with the PlanOut editor:

whitelist = @{'US': 1, 'UK': 1};
if (whitelist[country] != null) {
  result = "a";
} else {
  result = uniformChoice(choices=['b', 'c', 'd'], unit=country);
}

Basically, I just create a dictionary containing (as keys) the values I want to whitelist and set their (dictionary) values to any (non-null) value. This seems like it works, and is probably more performant than having to iterate through a list to check for existence of an element, especially if the list is very long. One could argue that this benefit outweighs the awkwardness of having to use a dictionary with dummy values just to check for simple existence, but seems like a relatively minor issue.

If this seems reasonable to recommend as an approach for this situation, I could make a pull request to update the documentation at some point. Or, does anyone have better ideas in mind?

Make it easier to enable / disable exposure logging

The PlanOut language has a return construct that allows experimenters to programmatically enable / disable exposure logging. For example, an experiment in which 20% of the users are eligible and are randomized into A and B can be written as:

eligible = bernoulliTrial(p=0.2, unit=userid);
if(!eligible) {
  return false;
}
variant = uniformChoice(choices=['a','b'], unit=userid);

However, many users of PlanOut do not use the PlanOut language and instead use a native API. Modify the PlanOut reference implementation so that that the return value of the assign() method determines logging. This would involve checking the return value of assign; if it is None (so that no return was triggered), in_experiment should be set to True; otherwise the truthiness of the return value should be used to keep track of the exposures. These changes might also be integrated into PlanOut.js (cc @rawls238 ) and the Go implementation (cc @tsujeeth)

Logging solutions for larger experiments?

Hi I'm wondering are there any recommendations/best practices on logging solutions for larger experiments, like ELK stack?

node command line interface not outputting json

Docs say node planout.js foo.planout should output the serialized json, but it currently outputs nothing. It appears the last working commit for this feature was 28603fe; broken starting in e064729.

Also .. how does switch case work ?

It looks like the following should be good for parsing, but jijson wont parse it

chances are I am reading the grammar wrong.

# Test Switch cases

a = unitId;

switch {
    case a == 10 then 
        b = a;
}

Interpreter and Experiment Demo

I downloaded this repo, and used pip to install planout
Then I ran demo_experiments.py from the terminal. I get this traceback:

Thomass-MacBook-Pro:demos Dalton$ python demo_experiments.py

Demoing experiment 1...
using simple_experiment_examples...
[(10, None), (1, 64), (10, 320), (1, 16), (10, 320), (1, 64), (1, None), (1, None), (1, 32), (10, None)]
using interpreter_experiment_examples...
Traceback (most recent call last):
  File "demo_experiments.py", line 35, in <module>
    demo_experiment1(interpreter)
  File "demo_experiments.py", line 7, in demo_experiment1
    print [(e.get('group_size'), e.get('ratings_goal')) for e in exp1_runs]
  File "/usr/local/lib/python2.7/site-packages/planout/experiment.py", line 24, in wrapped_f
    self._assign()
  File "/usr/local/lib/python2.7/site-packages/planout/experiment.py", line 63, in _assign
    self.assign(self._assignment, **self.inputs)
  File "/Users/Dalton/Documents/Projects/Planout/planoutGit/demos/interpreter_experiment_examples.py", line 18, in assign
    params
TypeError: __init__() takes at most 4 arguments (5 given)

I have no idea what the bug is here. I'm assuming that since I did noting but run the provided code, there is a bug in provided demos that will be obvious to someone with more experience with them.

No mapping between project git releases/tags and PyPI versions

There doesn't seem to be a clear way to know at what points on the code correspond to what versions got released to PyPI (for example, see the historic list at https://pypi.python.org/simple/planout/). This could be helpful to enable integrators of the framework to know exactly what code it is that they are using, and is a very common practice.

Could perhaps consider using git releases/tags for this in the future? (Even better would be to retroactively backtag the previous releases, but it's possible that information might have already been lost.) For example: https://github.com/kennethreitz/requests/tags and https://github.com/kennethreitz/requests/releases.

Assignment unaffected when salt set in SimpleNamespace

Looks like we might have a bug coming from line 173 in SimpleNamespace - if the Assignment is only generated in the Experiment's constructor, then it will not take into account the extra salt coming from the namespace.

Possible solution: re-set the Assignment's experiment_salt member whenever the experiment's salt setter is called?

Javascript Version

Are you aware of any JS versions of Planout, whether fully fleshed out or in flight? Wanted to check before I started an implementation. Thanks!

Inconsistencies for dictionaries

nvm

Make installable via package manager

Ideally make PlanOut installable via pip.

Pausing an experiment

Just wanted to get your thoughts on how to pause a running experiment/namespace. For me, pausing would mean that if the primary_unit has already been exposed, subsequent calls would continue to return the same exposure. However if this is the first time we've seen this primary_unit, they would get the default experiment while we're paused.

This would need a database in order to record the primary_units allocation for a given experiment/namespace.

Here is some sample code to illustrate what I mean:

from planout.experiment import SimpleExperiment
from operator import itemgetter

class PauseableNamespace(SimpleNamespace):
  paused = False

  def get_segment(self):
    allocated_segment = super().get_segment()

    # this value can also be hashed to make persistence easier
    primary_unit = itemgetter(*self.primary_unit)(self.inputs))
 
    # assume we have a `find_by` method that will query our db
    recorded_segment = find_by(primary_unit=primary_unit, namespace=self, segment=allocated_segment)
 
    if recorded_segment is None and self.paused:
      # First time we've seen this `primary_unit`. Since namespace is paused ensure default experiment is assigned.
      return -1

    # assume we have a `create` method that will insert a record into our db
    # record allocation
    create(primary_unit=primary_unit, namespace=self, segment=allocated_segment)

    return allocated_segment

Then my concrete namespace classes would inherit from this PauseableNamespace class.

Thoughts?

Expand documentation on return values

@rawls238 made some improvements to the logging documentation, but there should probably be an entire subsection on return values, noting that (1) it is common for experiments to include some kind of eligibility check, and that you should always return false if users fail the eligibility check, so that they don't get logged (2) returning false is also useful when you are dropping experimental conditions

External service example

This might include some code example whose assign() method includes some kind of check to an external service, e.g.,

country = get_user_country(userid)
if country != '':
  params.feature_enabled = false
  return false
params.feature_enabled = ...

Dropping arm example

Suppose you are running an experiment that tests 4 difference prices for a trade in a prediction market:

params.trade_price = uniformChoice(choices=[0.10, 0.25, 0.50, 0.99], unit=userid)

and it turns out that 0.99 performs very poorly, If we were to simply change the experiment to say,

params.trade_price = uniformChoice(choices=[0.10, 0.25, 0.50], unit=userid)

Doing this can have a number of negative effects: (1) it will reshuffle all users (2) in some cases (particularly with weightedChoice), doing this can create carryover effects, where users get re-shuffled in non-random ways.

One solution would be to simply replace 0.99 with your best guess of what works best (say it's 0.5):

params.trade_price = uniformChoice(choices=[0.10, 0.25, 0.50, 0.50], unit=userid)

This would break randomization in your experiment because potential carryover effects from those previously in the 0.99 condition could bias your estimates from the 0.50 condition.

To prevent this from happening, you can move all users who were previously in the 0.99 condition over to what you presently believe works best, and then tell PlanOut not to log the outcomes from those users:

params.trade_price = uniformChoice(choices=[0.10, 0.25, 0.50, 0.99], unit=userid)
if params.trade_price == 0.99:
  params.trade_price = 0.5
  return false

PHP version

Hi,
In the README.md you refer to the PHP version (should be in php/), but it is not there. Can you point me to the correct location?
Thank you,
Janos

unit test failures for python reference implementation

When I try running the unit tests for the Python reference implementation of PlanOut, I get:

ebakshy:test ebakshy$ python test_interpreter.py
Traceback (most recent call last):
  File "test_interpreter.py", line 11, in <module>
    from planout.interpreter import Interpreter
  File "//anaconda/lib/python2.7/site-packages/planout/interpreter.py", line 13, in <module>
    Operators.initFactory()
  File "//anaconda/lib/python2.7/site-packages/planout/ops/utils.py", line 26, in initFactory
    "map": core.Map,
AttributeError: 'module' object has no attribute 'Map'

@felixonmars , did you ever encounter this issue when you were refactoring things for python3 compatibility?

Namespace.get_segment bug

It appears the choice of segment is [0 .. num_segments] (inclusive) whereas the valid segments should be in range of [0 .. num_segments-1].

Using PlanOut with a CDN

Hi,

I'm asking here as I didn't manage to find alternative discussion groups or support channels for PlanOut. We're looking to revamp our approach to A/B testing, and we are concerned that our use of a CDN can result in caching tests, and our origin not getting the relevant data. There's a lot to learn in the area, and we're getting there slowly, but for now, we have one basic question - does PlanOut play well with CDNs?

I hope the question is sensible enough, please let me know if I can clarify it / make it sensible.

Antony

Unit test namespaces

Versioning and k-v units

Cheers for opening this up and sharing your work. It’s lovely to see such a succinct implementation of these powerful concepts.

A couple of questions came up while I was looking at implementing a PlanOut variant:

1. How do you handle versioning of the core specification? If you were to add a new operator, is that captured anywhere in the JSON specs that are passed around?
2. Why are units to operators used as a list instead of as a k-v map? Lists seem like they’d be easier to mess up in client implementations by accidentally switching the order. Do you see a downside to extending things to include hashes of strings like user_signup.my_exp.button_color.{"userid":"42"}?

Thanks much.