Versioning and k-v units

[rcs]

Versioning and k-v units

[eytan]

Thanks @rcs.

1: This sounds like an interesting question but I am not sure exactly what you are asking about here. Is there a particular concern that you had?

2: This is a good suggestion, and something we've gone back and forth on when we were first designing PlanOut. So instead of writing,

recommend_page = bernoulliTrial(p=0.99, unit=[userid, pageid]);  # (1)

one would write,

recommend_page = bernoulliTrial(p=0.99, unit={"userid": userid, "pageid": pageid}); # (2)

Or alternatively, write PlanOut code that reads like (1), but underneath the hood the compiler would instead generate units arguments that are dictionaries, so that unit=a is sugar for unit={'a': a}, and unit=[a,b] is sugar for unit={'a': a, 'b': b}.

We would then sort the unit dictionary by the keys and serialize it as e.g., "pageid:2.userid:9".

There are a few reasons we chose with the list version over the dictionary version. First, the former is easier to read and write (especially for non-programmers who may not be familiar with dictionaries/maps). Second, the dictionary version has this funny behavior of salting each unit by the unit's variable name, which could cause problems if a developer wanted to start changing around an experiment definition. Because if this, we felt that the list-version ends up being a lot more transparent and less error prone than the dictionary-based version.

[rcs]

Regarding 1:

If you're running a test producer (say the React web interface), that serializes experiments out to JSON form (or ships out the planout language specs?) and a number of consumers -- say a backend in PHP, a front-end in Javascript, a mailing program written in Go, there are a number of different code bases, of different types, that could have different versions of planout interpreters in them, and all be trying to bucket users into the same experiment segments.

If I had a version of a Go library being used since before the "round" operator was added, what's the right thing to do? Is there a way of knowing that the experiment spec isn't one I can run? Do I know that I should skip the test?

If in a future version of PlanOut, the definition for round() changes from the current rounding method to stochastic rounding a given number of decimal places in seeded by the hash, or changes to round based on an arbitrary-precision library instead of implicitly the 64bit float implementation, how do I know that the Go version is now out of sync, and will be returning different results?

If there's a case where the definition of the hash did change from 60 bits to 48, how could I handle that?

These are the kinds of things I'd expect to see managed by versioning the PlanOut 'protocol', and implemented at a test or multi-experiment level when loading.

[eytan]

Hi @rcs, at this point the Python implementation is the reference implementation, and I have so far reviewed all ports of PlanOut to other languages. So far, all ports have basic unit test coverage that match the reference implementation, and must produce random assignments that are consistent. I am a little skeptical that a full technological solution with protocol versions are necessary, and I would strongly favor a social solution, such as setting up a mailing list or Facebook group for discussing any breaking or major changes to PlanOut among maintainers of each implementation.

I am glad you pointed out the issue with round(), and I like the idea of having a more concrete standard. If you have any proposals for how to do a lightweight version of protocols (or better social solutions, involving standardized point releases), I'd love to hear them.

[eytan]

Closing this task out because it's not a behavior I would consider changing. Feel free to send me a PR or email me (my github user name at fb dot com) if you have any ideas for a concrete technological solution to the versioning. Also, FWIW, in the future we are going to have a standardized, cross-platform test suite, which I hope to make more exhaustive. Hopefully this should provide sufficient coverage for cases where other implementations of PlanOut don't match the reference implementation.