mixcloud/django-experiments

Django AB testing module

django-experiments's Issues

Control buttons in results page

Should be able to change the status/edit the experiment once actually within the results page.

Should come into effect with #12.

Goals and enrollments

Would it make sense for a goal to be only incremented once per enrollment?

Nexus index styling

Improve the nexus front page styling. More information would also be useful here.

Nexus dashboard not showing active selective experiments

The nexus dashboard currently only shows experiments that are toggled to global. Need to add support for selectively activated experiments too.

Conditions cannot add on new experiment

When a new experiment is added, the page needs to be refreshed before a condition can be added

Significance bugs

i.e. N/A

Was on TODO, can't really remember what this is about =P

No License

Need to add a license (see MIT) to the root of the directory.

Setting conditions before switching to selective

Cannot set conditions before switching to selective mode as conditions are hidden unless in selective mode.

Timing support

start_time and end_time need to be populated automatically. What about scheduling the experiment to end at a certain time?

URL/Slug Base Partitioning.

Cannot remember what this was, but it is on the TODO list.

"url/slug base partitioning, for object/page testing. Googlebot based testing"

States for alternatives

Add enabled/disabled states to specific alternatives within an experiment.

One Goal Per Enrollment

Based on the chi squared test, a goal should only be incremented once per enrollment.

Alternative template hits

When defining alternatives in a template, each alternative hits the server separately. Could be an issue with heavy traffic.

Condition Behaviour

Investigate the way conditions are inclusive/exclusive of one another.

Currently, conditions are inclusive of one another within a condition_set but exclusive between sets.

Storage Engine - Django ORM, Redis

Cannot remember what this is, but it was on the TODO list.

Control column is the same for all goals

in nexus_modules.py line 165.

Goals not tested

Goals have not been tested in the last few commits.

Tidy up redis counters

Tidy up redis counters when the experiment is deleted. Will currently be stored indefinitely.

Attach a signal and use KEYS/DEL commands: KEYS experiments:<experiment_name>:*

Enrollments prevent experiment caching

Enrollments contain experiment foreign keys, and so when experiments are called through the enrollment, i.e.

utils.py line 153: if enrollment.experiment.state == DISABLED_STATE:

the experiment cache is not used. Should replace with the experiment_name, and use the experimentmanager cache to retrieve the experiment.

sorting issues

start date and end date sorting do not currently do anything.

Date Inconsistancy

Dates style is different when the nexus index page is loaded and when a new experiment is added. This is due to differences in the way the data is rendered. On page load, the QuerySet is rendered into the template directly via django. When a new experiment is added, the data is serialized server side, and de-serialized client side. The de-serialization causes the date to take on a slightly different format to when the data is rendered directly.

Delete Experiment Performance

Deleting experiments and enrollments causes major performance issues many db records are deleted at once. In the short term:

• Make delete experiment performant
• Remove old enrollments/sessions

In the long term, the way enrollments are handled needs to be looked into.

end_date never assigned

end_date is never assigned a value. Should be done when an experiment is disabled.

Goal Control

Add the ability to modify/reset goal values through nexus.

Possibly even relocate goals from settings.py to redis.

Using gargoyle as a dependency

Many implications. Needs to discuss.

Code needs tidying

It is very messy.

"Just Now" when just created.

Created in fa82496c2608187b6a55206094ea72458be90e72, which solved #27. "Just Now" is only used to prevent the server from mysteriously hanging when adding a new experiment. It is something to do with returning an experiment.to_dict() which contains a start_date of type DateTime.

Cycle counter within for loop.

Cycle counter does not reset when being used within a for loop, making it difficult to create alternating column classes when dynamically generating them in the results.html table.

Currently using -{% if forloop.counter|divisibleby:2 %}even{% else %}odd{% endif %}. Waiting for a patch to fix this.

See http://stackoverflow.com/questions/459161/alternate-row-coloring-in-django-template-with-more-than-one-set-of-rows

Quick Show All Goals

Add a link/button to quickly show all available goals on the results page.

Check relevance of index.html code

Line 175 (as of time of writing). Need to clarify how this code is used and whether it is now depreciated.

Improve Results Styling

Give results CSS some much needed beautification.

Add status and edit buttons.

Results goal toggling

Add JS to toggle goals from within the results table. Similar to condition toggling?

Conditions and Sessions

If no conditions are met, the user does not get a session for the experiment. If then a condition is added which they match, they will receive session data for the experiment.

Their session data may be different before and after they meet the condition, and so they could jump from the control to another alternative. Is this correct?

Invalid IP address in condition hangs

If an invalid IP address is used in an IP address condition, the server will not send a response. The error is being caught, however, as it appears in the console log.

Goal hits are not unique to experiment

Every experiment will record a every goal hit, even if the hit is intended to come from another experiment. This can cause very inaccurate data.

Closely related to #8.

ModelDict cache should be invalidated upon experiment change

May already be implemented. Need to check.

If cache is not invalidated, then experiments will not update for users until cache is reset.

Some conditions not working

Conditions do not seem to be functioning correctly when only a single "exclude" condition is set.

Tested on not anonymous and not username.

autocreate using modeldict manager

experimentmanager should autocreate the experiment if it doesn't exist. This will prevent keyerrors.

Add logging

Log the following to file

• users are added to an experiment, and which alternative
• goal hit (first time and multiple)
• confirmed human
• anything else of value....

Edit experiment within results

Cannot currently edit the experiment settings (bar conditions) from within the experiment results view. Can either overlay edit experiment form or follow design similar to that suggested in #39.

Caching at runtime to reduce hits

Caching or experiment model at runtime to reduce db hits (see gargoyle and modeldict)

Collapsing/Expanding in Results

The results page should support the collapsing and expanding of alternatives.

This could cause issues with the odd/even classes of columns. Could prevent this issue by shrinking the alternative from 3 columns to one column.

Cannot add conditions in new results page

To do with JS and template.

Caused in 9acb738438a174a2d6f2ed686d6da7e905024f16

Duplicate Condition Deletion

If there are duplicate conditions and the user tries to delete one of them, all will be deleted. The delete function is also ignorant to the excluded variable.

Changes to remove_conditions() in nexus_modules.py:

• Add exclude as a criteria, also make sure to POST it in js.
• Only delete the first entry if multiple are found matching the criteria.

Zero Conditions

Should we assume that the experiment state is global if there are no conditions?

Specific Goal Tracking

Allow specific goal tracking in results view.

Goals should not increment if experiment disabled

Forgot to git add .

Have forgotten to git add . in the last few commits so there is a whole load of missing code. Fixing now.

Project Name

The project name needs to be changed. Causes some clarity issues with the code, as experiments is used to loosely.

Post ideas in the comments ;)

Toggling experiment status causes page to hang

Was caused in fa82496c2608187b6a55206094ea72458be90e72 when fixing #27. Caused by the use of datetime data in the start_date field.

Note, still works after a page refresh. Check JavaScript and that functions are returning correctly.