How to use SQL backend to store experiments when using Service API

[facebook]

How to use SQL backend to store experiments when using Service API

[lena-kashtelyan]

Hey, @hyzhak, glad to hear that it will be useful; it is almost done!

[lena-kashtelyan]

@dkatz23238, thank you so much for this detailed feedback!

Re: docs on saving experiment –– that will be in soon! We didn't add it yet because we were not encouraging accessing experiment through the service API just yet, so that will change shortly.

Re: allowing to grab some plotting platform-agnostic data for plots –– understood, great idea, thank you!

Re: unique name for altered experiments –– yes, experiments do need to have unique names for storage, we'll make that more clear in the docs. Just to make sure: you can alter the optimization config on an existing experiment, but if you are creating a new experiment, it will need a unique name, even if it's an exact copy of an existing experiment.

[lena-kashtelyan]

Hey, @dkatz23238! Just to clarify, are you looking to just save the underlying experiment in the service API, or are you looking to save the full state of optimization you're doing through service API, so that you can continue it at a later time (this latter option involves saving both the experiment and the state of the optimization, e.g. which model was being used, what its state was. etc.)?

Both of those use cases are currently in the works. However, if you are looking for the former, there is a workaround for now: you can save the experiment in the same way you would save it in the developer API (thorough save_experiment(experiment=ax_client.experiment). If the latter, that functionality is under development and will be available in later versions of Ax.

[dkatz23238]

I was looking for the best practice you gives you to store results for later inspection. The ax_client.experiment should work for my use case. It was just a bit confusing at first since the developer and loop API give you an explicit experiment object.

I am trying to store experiments in a backend to later query for visualization purposes using something like Power BI that would be the defacto tool we use in my organization.

Again, thanks for your help Lena!

[lena-kashtelyan]

@dkatz23238, that makes total sense, thank you for pointing that out! The current workaround works for you for now, then?

Just out of curiosity, do you plan to use Ax visualizations or your own separate visualizations? Would it help if the service API provided an ability to directly retrieve (as plotly configs of AxPlotConfig-s) the contour plot of response surfaces and trace of optimization on objective?

[dkatz23238]

My use case for having a db is to store many experiments and later visualize them among a small team sharing code. I don't think I will be using plotly because in my case we tend to stick to other solutions. From what I have seen so far I can easily join among the arm_v2 and metrics_v2 table to create the plots I need. This also makes it easy to create web visualizations using javascript and html.

I like the normalization work that has already done, thats why I wanted to test these features out instead of just storing to json or to a json style db like mongo.

The ax.experiment suggestion definitely worked for me however maybe the documentation can be a little bit more clear as there is no direct tutorial on storing an experiment. I wanted to make sure that sqa_store.db.create_all_tables(engine) did not delete any tables that already existed.
In the end my code looks something like this:

import ax.storage as axst

url = "mysql://root:password@localhost/axdb"
axst.sqa_store.db.init_engine_and_session_factory(url=url)
engine = axst.sqa_store.db.get_engine()
axst.sqa_store.db.create_all_tables(engine)
axst.sqa_store.save(experiment=ax.experiment)     

Regarding your last question I do think this would be useful for plotly users, but maybe there can be a simple "framework" agnostic way to return only the data needed to produce a series of recommended plots.

What I plan on doing is just retrieving the data from SQL into a pandas dataframe for exploratory analysis.

Just an additional note:

Seems like every experiment stored in a mySQL db should have a unique name even if there have been changes to say the parameters or objective function. I dont know if this is documented anywhere but I have not found it.

My quick fix is the following

isonow = lambda : datetime.datetime.now().isoformat()

ax.create_experiment(
    name="GaussianProcessRegression-%s" % isonow(),
    parameters=PARAMETERS,
    objective_name="mean_square_error",
    minimize=True)

[hyzhak]

I've faced similar problem. I need to preserve state of experiments on a long run of search (Service API), tried to use ax.storage.sqa_store.structs.DBSettings but constantly getting :

NotImplementedError:
Saving and loading experiment in `AxClient` functionality currently under development.

So I can't wait to finally get that functionality it will be extremely useful

[lena-kashtelyan]

There is now an ability to use SQA storage for the Service API. @hyzhak, @dkatz23238, you can make use of it by specifying DBSettings here: https://github.com/facebook/Ax/blob/master/ax/service/ax_client.py#L103.

Not closing this issue yet as I would still like to add an example of how to use the DBSettings to the Service API, but just letting you know!

[hyzhak]

@lena-kashtelyan hurray! Thank you, will try soon!