Comments (11)
Hello again, @quanvuong!
-
You can use
render(ax_client.get_optimization_trace())
to see the objective values so far (API reference). We will add parameter values on hover to this plot, to make it easier to understand what the most promising parameters so far are. The resulting plot currently will look something like this:
. -
Length scale / sensitivity analysis –– in the works, thank you for bringing it up!
-
Extracting visualization plot –– we will look into a more easily serializable way of providing the plots, thanks for the feedback!
-
Will add parameter values on hover to plot shown in 1), would that, in combination with the contour plot, be sufficient for the purpose of detecting whether the search space needs adjustment?
from ax.
Hello, @quanvuong! Just for context, which one of our APIs are you using and would want to see this functionality implemented in?
from ax.
Hi!
I'm using the Service API. The Service API allows me to manually logs the hyper-parameter values and their corresponding function evaluation values. But it would be nice if this feature was already implemented.
from ax.
@quanvuong, just so that I understand, does the following code not achieve this purpose? What should we add?
from ax.service.ax_client import AxClient
from ax.utils.measurement.synthetic_functions import branin
from ax.utils.notebook.plotting import render
ax = AxClient()
ax.create_experiment(
parameters=[
{"name": "x", "type": "range", "bounds": [-5.0, 10.0]},
{"name": "y", "type": "range", "bounds": [0.0, 15.0]}
]
)
for _ in range(6):
params, idx = ax.get_next_trial()
ax.complete_trial(idx, branin(*(params.values())))
ax.get_best_parameters()
render(ax.get_contour_plot())
Just for ease of reference, this is what it looks like executed in a notebook:
from ax.
Thanks!
For example:
-
The contour plot does not show the function evaluation values, the corresponding hyper-parameter in a ranked manner. This would be useful during optimization to get a sense of what is the most promising hyper-parameter values so far.
-
Length Scale or some sort of sensitivity analysis. Before running BayesOpt, I might not know what the important hyper-parameters are and what hyper-parameters the performance is relatively insensitive too. It would be super useful to show how sensitive the performance is with respect to each hyper-parameter. This would inform subsequent experiments and understanding.
-
I might run BayesOpt inside a docker image or on a server. So it's not easy to extract out visualization plot. In this case, textual logs would be useful as well.
from ax.
Another feature that would be helpful is:
- Viewing the hyper-parameter values of the best function evaluations so far. This helps to see if the optimized hyper-parameter values hit their respective bounds, meaning that the bounds for some of the hyper-parameters are too tight. This usually means that better hyper-parameter values can be found by loosening the bounds the hyper-parameter values are searched over.
from ax.
yes, that would be great. Thanks!
from ax.
Parameter values on-hover are now part of the optimization trace plot for the Service API and more textual logs are now enabled in the latest stable version, 0.1.6. Examples in the tutorial: https://ax.dev/tutorials/gpei_hartmann_service.html.
Sensitivity analysis plot and extraction of plots in serializable way are still in the works.
from ax.
We have a pending change to the service API:
render(ax_client.get_feature_importances())
Will show a ranked bar chart of feature importances for different metrics.
from ax.
Also, @quanvuong, while we consider a slightly more elegant option you can serialize and visualize plots as follows:
from ax.
I believe this should address all the issues brought up in this task: textual logs and parameter values on hover (included in the latest stable release), sensitivity analysis plots (included on master), and a workaround for serializing the plots. Closing the issue, but feel free to follow up, @quanvuong!
from ax.
Related Issues (20)
- attach trials HOT 8
- Multi-task BO with Service API HOT 2
- constrains HOT 17
- Question : Generation 12 trials HOT 8
- Question: Multi-Task Multi Objective HOT 1
- MOO not respecting nonlinear constraints HOT 1
- Problem with Fixed parameters if nonlinear_inequality_constraint is imposed
- Safe optimization in the Service API HOT 4
- The same point is evaluated multiple times during Integer Optimization with BO. HOT 5
- ax_client.generation_strategy.trials_as_df HOT 9
- Managing Objective Function Evaluation Failures in Ax for MOO HOT 6
- [GENERAL SUPPORT]: Managing Objective Function Evaluation Failures in Ax for MOO HOT 3
- [GENERAL SUPPORT]: Using qNegIntegratedPosteriorVariance HOT 3
- [GENERAL SUPPORT]: Reference point for multi-objective bayesian optimization HOT 4
- [GENERAL SUPPORT]: Adjusting search space or accommodating out-of-bounds initial data HOT 19
- [GENERAL SUPPORT]: Manual configuration, HOT 1
- [Bug]: Custom metric issue HOT 4
- [GENERAL SUPPORT]: CI_Level Paretofrontier
- [Bug]: Large sample time increase in ax-platform >= version 0.3.5 HOT 6
- [GENERAL SUPPORT]: Reference Point for Multi-Objective Bayesian Optimization HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ax.