Comments (9)
Your response summarizes very well why we didn't implement special behavior out of the box, since 1) attached trials may be anything or nothing, 2) unless trials were generated by Ax and for an experiment with the same setup (search space + evaluation setup), it may be hard to tell whether they are a reasonable sample. However, the repeated trials issue comes up only if those trials were actually generated by Ax, so it we will still look into possibly adding a fix, since in those cases the attached trials do constitute a reasonable sample.
Also, I just realized that this may not be obvious: those attached trials are actually used as training data for BayesOpt, so information in them is used to improve optimization. The reason why the initial 5 trials are the same is that they are generated quasi-randomly, but starting with the 6th trial generation, Bayesian optimization kicks in and utilized the data from the attached trials.
from ax.
@aagarwal1999, you could actually achieve that behavior by passing in a custom generation strategy to AxClient
. There are two kinds of generation strategies you could use:
-
Generation strategy with only the Bayesian optimization step, without the quasi-random initialization. You would just need to make a generation strategy, similar to that returned by
choose_generation_strategy
(referenced in #151) but with only one step, theModels.GPEI
step. Note that if you went with this solution, you would need to make sure you do always attach and complete a few trials before generating new ones fromAxClient
viaget_next_trial
(since the model you are using in this case needs some training data right away). It's also only really good to do this if the previous trials were generated through Ax (or in some other principled way), so that they are a reasonable sample of the search space. -
Generation strategy, in which the initial quasi-random trials start later in the Sobol sequence (so they wouldn't be repeated). Your generation strategy would still be similar to that returned by
choose_generation_strategy
(referenced in #151) but with an additional kwarg here,init_position
=N, where N is how many trials you will be attaching. You could also reduce the number of Sobol trials done in this case to, say, 2, since you know you are attaching trials, previously generated by Ax, so there will be enough training data (generated in a principled way) for the BayesOpt model to learn.
Do these solutions help?
We are also currently looking into making sure that the trials, which overlap with the ones attached by users, are not suggested again quasi-randomly, and I will update this thread if we decide to implement a fix for it. It's just never come up before as folks weren't fixing the seed, so thank you for bringing it up : )
from ax.
For now the random seed work around I suggested will suffice, since the attached trials could be anything (or nothing) and I do not know how to tell if the given parameters are a "reasonable sample" of our search space without diving too much into Ax's codebase. But please let me know if that fix is implemented.
from ax.
Hi @aagarwal1999, if you are adding any additional datapoints to Ax, it should be from the exact same task—otherwise it will really throw off the GP. You can always use custom models with Ax, so if you wanted to use a different mean function of HP prior, you could do that. We also have an example of meta-learning in BoTorch, which could be adapted to work with Ax. With this algorithm, you can use any number of possibly related runs.
Another thing to note is that if you did want to use any prior experiments for improving the efficiency of your optimization, you'll probably need way more than 5 previous points, since one generally needs at least at least 2-3 points per dimension to start getting a sensible GP model fit.
from ax.
@aagarwal1999, are you fixing the random seed?
from ax.
from ax.
Yes I am fixing the random seed for reproducibility.
Below is the example I ran. You can see even when I attach values of 4, 18, 2 before I run get_next_trial
and rerun the experiment, the trials that come out are also 4, 18, and 2 for the first three trials. This happens with or without me setting torch.manual_seed before every call of generate_trial.
from ax.service.ax_client import AxClient
from ax.service.utils.dispatch import choose_generation_strategy
import torch
ax_client = AxClient()
name = "test_ax"
parameters = [{"name": "param_1", "type": "range", "bounds": [1, 100], "log_scale": True}]
objective_name = "objective_1"
minimize = True
ax_client.create_experiment(name=name, parameters=parameters, objective_name=objective_name, minimize=minimize)
ax_client.generation_strategy = choose_generation_strategy(search_space=ax_client.experiment.search_space, random_seed=42)
params = []
for _ in range(3):
torch.manual_seed(1000)
param, trial_id = ax_client.get_next_trial()
params.append(param["param_1"])
print(param)
ax_client = AxClient()
ax_client.create_experiment(name=name, parameters=parameters, objective_name=objective_name, minimize=minimize)
ax_client.generation_strategy = choose_generation_strategy(search_space=ax_client.experiment.search_space, random_seed=42)
for param in params:
print(f"Attaching trial with param value {param} and result {param * 2}")
new_param, trial_id = ax_client.attach_trial({"param_1": param})
ax_client.complete_trial(trial_id, raw_data=param * 2)
for _ in range(3):
torch.manual_seed(1000)
param, trial_id = ax_client.get_next_trial()
print(param)
from ax.
The reason why you see this behavior is that the first 5 trials are generated using Sobol sequence (quasi-randomly), so when you fix the random seed, the first few trials still come out the same. Do you need both the reproducibility and to be able to use the trials from the previous optimizations?
from ax.
Not really, I can work around it by having the user set a seed as a part of the configuration. It just seems to me that if I am trying to give a prior to ax in hopes of improving optimization time, the first five trials should be generated from a more optimized search space based on the information given to the client.
Anyway, if this is not a big priority right now, this is not a big deal. Thanks for your help!
from ax.
Related Issues (20)
- Hierarchical Search Spaces with Multiple Independent Search Spaces HOT 3
- Services API :×1-1.5*×2>=0 HOT 4
- primary_objective and secondary_objective HOT 1
- Comparison of multi-objective acquistion functions HOT 14
- plot_pareto_frontier HOT 6
- attach trials HOT 8
- Multi-task BO with Service API HOT 2
- constrains HOT 17
- Question : Generation 12 trials HOT 8
- Question: Multi-Task Multi Objective HOT 1
- MOO not respecting nonlinear constraints HOT 1
- Problem with Fixed parameters if nonlinear_inequality_constraint is imposed
- Safe optimization in the Service API HOT 4
- The same point is evaluated multiple times during Integer Optimization with BO. HOT 5
- ax_client.generation_strategy.trials_as_df HOT 9
- Managing Objective Function Evaluation Failures in Ax for MOO HOT 6
- [GENERAL SUPPORT]: Managing Objective Function Evaluation Failures in Ax for MOO HOT 3
- [GENERAL SUPPORT]: Using qNegIntegratedPosteriorVariance HOT 3
- [GENERAL SUPPORT]: Reference point for multi-objective bayesian optimization HOT 4
- [GENERAL SUPPORT]: Adjusting search space or accommodating out-of-bounds initial data HOT 19
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ax.