Looking for suggestions for the objectives data

[lchen4snap]

Looking for suggestions for the objectives data

[mgrange1998]

Hi, thank you for reaching out with this question.

Ax's bayesian optimization does not perform well for non-stationary data, such as noticeable differences between day and night.

Here is another issue which asks a similar question, so please read through the discussion for more context #2342

One quote from the thread:
"Note that AB tests cause some non stationary, in that treatment effects
change over time. I recommend making sure each batch runs for enough time
to “settle down”, and using the same number of days per batch. "

So rather than changing the "raw_data" to account for noise, another approach is to have batch trials run for longer than a day so that each batch can run across the non-stationary cycle.

I'll cc @mpolson64 and @eytan for more context. Let me know if this helps and if you have any other questions.

[lchen4snap]

Thanks @mgrange1998 ! Running the batch for longer than a day makes sense for this case. From the thread you pointed, it looks common that other folks are also using pct change for the metrics.

[Balandat]

Relativizing the data w.r.t. the control group often gets rid of a bulk of the non-stationarity - at least that part that is attributable to the DGP and not non-stationarity of the treatment effect itself. If you do have observed variances you probably want to account for that in the relativization (we use the Delta method for this).

Another thing that can help is to use the time of day or day of the week as an additional covariate - unfortunately that's not straightforward to do in the current Ax setup (though we're working on making that easier). cc @sdaulton