evhub/bbopt

Black box hyperparameter optimization made easy.

bbopt's Issues

Prevent regenerating the model with each run

It'd be nice to have a way to save the generated model so it can be updated instead of just regenerating it.

I wish i'd seen this...

I rather wish I'd noticed this before writing humpday.

[Errno 9] Bad file descriptor

Hi,

running the code from the command line. Tried using json - did not help. However, if I comment self._load_data(), things seem to work. The question is: does the comment break the optimization process?

Thank You!

optimize.py
def reload(self):
"""Completely reload the optimizer."""
self._old_params = {}
self._examples = []
#self._load_data() <-------------------------------------------------COMMENTED
self.run(alg=None) # backend is set to serving by default

Traceback (most recent call last):
File "test.py", line 26, in
bb = BlackBoxOptimizer(file=file,use_json=True)
File "/home/b7066789/.local/lib/python3.6/site-packages/bbopt/optimizer.py", line 72, in init
self.reload()
File "/home/b7066789/.local/lib/python3.6/site-packages/bbopt/optimizer.py", line 78, in reload
self._load_data()
File "/home/b7066789/.local/lib/python3.6/site-packages/bbopt/optimizer.py", line 192, in _load_data
with Lock(self.data_file, "rb", timeout=lock_timeout) as df:
File "/home/b7066789/.local/lib/python3.6/site-packages/portalocker/utils.py", line 197, in enter
return self.acquire()
File "/home/b7066789/.local/lib/python3.6/site-packages/portalocker/utils.py", line 157, in acquire
raise exceptions.LockException(exception)
portalocker.exceptions.LockException: [Errno 9] Bad file descriptor

System:
CentOS 7

pip freeze
absl-py==0.2.2
anndata==0.6.6
args==0.1.0
asdf==2.1.0
ast2vec==0.3.8a0
astetik==1.9.5
astor==0.6.2
astropy==3.0.4
atomicwrites==1.2.1
attrs==18.2.0
backcall==0.1.0
bblfsh==2.9.13
bbopt==0.4.1
beautifulsoup4==4.6.3
biopython==1.72
bleach==1.5.0
boto==2.49.0
boto3==1.9.86
botocore==1.12.86
bs4==0.0.1
bz2file==0.98
cachetools==2.1.0
certifi==2018.8.24
chances==0.1.4
chardet==3.0.4
Click==7.0
clint==0.5.1
colorama==0.3.9
cycler==0.10.0
cymem==2.0.2
cytoolz==0.9.0.1
datasketch==1.4.1
decorator==4.3.0
dill==0.2.9
docker==3.5.0
docker-pycreds==0.3.0
docutils==0.14
dulwich==0.19.6
EasyProcess==0.2.3
en-core-web-sm==2.0.0
entrypoints==0.2.3
fa2==0.2
flake8==3.6.0
flake8-polyfill==1.0.2
flatbuffers==1.10
funcsigs==1.0.2
funcy==1.11
future==0.17.1
gast==0.2.0
gensim==3.7.0
geonamescache==1.0.1
google-api-core==1.4.0
google-auth==1.5.1
google-auth-httplib2==0.0.3
google-cloud-core==0.25.0
google-cloud-storage==1.2.0
google-resumable-media==0.3.1
googleapis-common-protos==1.5.3
GPUtil==1.3.0
graphviz==0.10.1
grpcio==1.10.0
grpcio-tools==1.10.0
h5py==2.7.1
html5lib==0.9999999
HTMLParser==0.0.2
HTSeq==0.10.0
httplib2==0.11.3
humanize==0.5.1
hyperas==0.4
hyperopt==0.1.1
idna==2.7
igraph==0.1.11
imageio==2.4.1
ipykernel==4.8.2
ipython==6.4.0
ipython-genutils==0.2.0
ipywidgets==7.4.2
jedi==0.12.0
Jinja2==2.10
jmespath==0.9.3
joblib==0.11
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
Keras==2.2.4
Keras-Applications==1.0.7
Keras-Preprocessing==1.0.9
kerasplotlib==0.1.4
kiwisolver==1.0.1
langdetect==1.0.7
lightgbm==2.2.2
livelossplot==0.2.2
llvmlite==0.23.0
louvain==0.6.1
lxml==4.2.1
lz4==2.1.0
mando==0.6.4
Markdown==2.6.11
MarkupSafe==1.0
matplotlib==3.0.2
mccabe==0.6.1
mistune==0.8.3
mnnpy==0.1.9.4
modelforge==0.7.0
modin==0.2.5
more-itertools==4.3.0
msgpack==0.5.6
msgpack-numpy==0.4.3.2
murmurhash==1.0.1
natsort==5.3.1
nbconvert==5.3.1
nbformat==4.4.0
netifaces==0.10.7
networkx==1.11
nltk==3.4
notebook==5.6.0
numba==0.38.0
numexpr==2.6.5
numpy==1.14.5
pandas==0.23.4
pandocfilters==1.4.2
parquet==1.2
parso==0.2.0
patool==1.12
patsy==0.5.0
pexpect==4.5.0
pickleshare==0.7.4
Pillow==5.1.0
plac==0.9.6
pluggy==0.7.1
ply==3.11
portalocker==1.4.0
pqdict==1.0.0
preshed==2.0.1
prometheus-client==0.3.1
prompt-toolkit==1.0.15
protobuf==3.6.1
psutil==5.4.8
ptyprocess==0.5.2
py==1.6.0
py4j==0.10.7
pyasn1==0.4.4
pyasn1-modules==0.2.2
pycodestyle==2.4.0
pyflakes==2.0.0
Pygments==2.2.0
pyLDAvis==2.1.2
pymongo==3.7.2
pyparsing==2.2.0
pysam==0.14.1
pyspark==2.4.0
PyStemmer==1.3.0
pytest==3.8.2
pytextrank==1.1.0
python-dateutil==2.7.3
python-igraph==0.7.1.post6
python-libsbml==5.17.0
python-pptx==0.6.9
python-snappy==0.5.3
pytz==2018.4
pyunpack==0.1.2
PyWavelets==1.0.1
PyYAML==3.12
pyzmq==17.1.2
qtconsole==4.4.1
radon==2.4.0
ray==0.6.0
redis==3.0.1
regex==2018.1.10
requests==2.20.1
rsa==4.0
s3transfer==0.1.13
scanpy==1.2.2
scikit-image==0.13.1
scikit-learn==0.20.1
scikit-optimize==0.5.2
scipy==1.1.0
seaborn==0.8.1
selectolax==0.1.8
semantic-version==2.6.0
Send2Trash==1.5.0
sh==1.12.14
shap==0.28.5
simplegeneric==0.8.1
simplejson==3.16.0
singledispatch==3.4.0.3
six==1.11.0
sklearn==0.0
smart-open==1.8.0
sourced-engine==0.6.4
sourced-ml==0.6.3
spacy==2.0.18
statistics==1.0.3.5
statsmodels==0.9.0
stocal==1.2
summa==1.2.0
tables==3.4.3
talos==0.4.8
tensorboard==1.12.2
tensorflow==1.12.0
termcolor==1.1.0
terminado==0.8.1
testpath==0.3.1
Theano==1.0.2
thinc==6.12.1
thriftpy==0.3.9
tmsc==0.1.5a0
toolz==0.9.0
torch==0.4.1
torchvision==0.2.1
tornado==5.1
tqdm==4.31.1
traitlets==4.3.2
typing==3.6.6
ujson==1.35
umap==0.1.1
umap-learn==0.2.3
urllib3==1.23
wcwidth==0.1.7
websocket-client==0.53.0
Werkzeug==0.14.1
widgetsnbextension==3.4.0
wrangle==0.5.1
wrapt==1.10.11
xgboost==0.72
XlsxWriter==1.0.5
xmlutils==1.4

The code:

X_train, Y_train, X_val, Y_val = data()

def run_trial():
"""Run one trial of hyperparameter optimization."""
# Start BBopt:
bb.run()
input_shape = 8208

h_n_num = bb.randint('num_neur',5,1000)
act = bb.choice('activ_func',['selu','relu','elu'])
num_lay = bb.randint('num_hidden_layers',0,10)
dout = bb.uniform("dropout", 0, 1)
lr = bb.uniform("init_learn_rate", 1e-5, 0.1)
bsize = bb.choice('batch_size',[8,16,32,64,128])

# Create model:
a = Input(shape=(input_shape,))
b = Dense(h_n_num,activation=act)(a)
b = Dropout(dout)(b)
for l in range(num_lay):
    b = Dense(h_n_num,activation=act)(b)
    b = Dropout(dout)(b)
output = Dense(1,activation='linear',name='out')(b)

model = keras.models.Model(inputs=a, outputs=output)
opt = Nadam(lr)
model.compile(optimizer = opt, loss=mse)

# Train model:
history = model.fit(x=X_train[:-70], y=Y_train[:-70],batch_size=bsize,epochs=1,
                          validation_data=(X_train[-70:],Y_train[-70:]),
                             verbose=0,
                             validation_split = 0.4,
                             callbacks=[EarlyStopping(patience=30), 
                                        ReduceLROnPlateau(monitor='val_loss', factor=0.5, patience=25,verbose=0)], 
                             shuffle=False)

train_loss = history.history["loss"][-1]
val_loss = history.history["val_loss"][-1]

bb.remember({
    "train_loss": train_loss,
    "val_loss": val_loss,
})

bb.minimize(val_loss)

num_trials = 5
result = []

for i in tqdm(range(num_trials)):
run_trial()
result.append(bb.get_current_run())
if len(result)>1:
[i['memo'].update(i['values']) for i in result]
temp = [i['memo'] for i in result]
pd.DataFrame(temp).to_csv('./transfer_learning/DL_optimization_reports/patch_weekly_5000_trials.csv')

ValueError: 16 is not in list

Hi，here is the problem when i using BBopt in my project.
Traceback (most recent call last):
File "I:/intention recognition/MyBBopt/testFile.py", line 167, in <module>
run_trial()
File "I:/intention recognition/MyBBopt/testFile.py", line 81, in run_trial
bb.run()
File "D:\Anaconda\lib\site-packages\bbopt\optimizer.py", line 255, in run
self.run_backend(backend, **options)
File "D:\Anaconda\lib\site-packages\bbopt\optimizer.py", line 242, in run_backend
self.backend = init_backend(backend, self._examples, self._old_params, *args, **options)
File "D:\Anaconda\lib\site-packages\bbopt\backends\hyperopt.py", line 132, in __init__ trial_list = examples_to_trials(examples, params)
File "D:\Anaconda\lib\site-packages\bbopt\backends\hyperopt.py", line 103, in examples_to_trials
for k, v in zip(sorted(params), make_features(ex["values"], params, fallback_func=lambda name, func, *args, **kwargs: NA, converters={"choice": lambda val, choices: choices.index(val), "randrange": lambda val, start, stop, step: val - start}, convert_fallback=False)):
File "D:\Anaconda\lib\site-packages\bbopt\backends\util.py", line 90, in make_features
feature = converter_func(feature, *args)
File "D:\Anaconda\lib\site-packages\bbopt\backends\hyperopt.py", line 103, in <lambda>
for k, v in zip(sorted(params), make_features(ex["values"], params, fallback_func=lambda name, func, *args, **kwargs: NA, converters={"choice": lambda val, choices: choices.index(val), "randrange": lambda val, start, stop, step: val - start}, convert_fallback=False)):
ValueError: 16 is not in list

i really confused，what's the meaning of 16? it doesn't exist in my project. Could you give me any advice?

[BUG]: `numpy.random.mtrand.RandomState` object has no attribute `integers` -> switch to `np.random.Generator`

Hey,

I found a bug, which is easy to fix, the hyperopt backend does not use the deprecated np.random.RandomState anymore and replaced it with np.random.Generator, this means the following line:

bbopt/bbopt/backends/hyperopt.py

Line 176 in 408e210

def setup_backend(self, params, algo=tpe.suggest, rstate=np.random.RandomState(), show_progressbar=False, **options):

Should be changed to:

    def setup_backend(self, params, algo=tpe.suggest, rstate=np.random.default_rng(), show_progressbar=False, **options): 

This is explained in the following issue on hyperopt:

hyperopt/hyperopt#838

Best,
Max

[FEATURE]: Run Id should be available in a process

While using bbopt each run process has its own run id, which is not available during each run.

It would be awesome if the run id can either be accessed via the BlackBoxOptimizer object or via an environment variable, which has to be then set for each process.

TypeError: Cannot cast array data from dtype('<U4') to dtype('int64') according to the rule 'safe'

Traceback (most recent call last):
File "bayesian_opt_DL.py", line 137, in
run_trial()
File "bayesian_opt_DL.py", line 89, in run_trial
bb.run()
File "/home/b7066789/.local/lib/python3.6/site-packages/bbopt/optimizer.py", line 107, in run
self.run_backend(backend, **kwargs)
File "/home/b7066789/.local/lib/python3.6/site-packages/bbopt/optimizer.py", line 95, in run_backend
self.backend = init_backend(backend, self._examples, self._old_params, *args, **kwargs)
File "/home/b7066789/.local/lib/python3.6/site-packages/bbopt/registry.py", line 105, in init_backend
return backend_registry[name](examples, params, *args, **kwargs)
File "/home/b7066789/.local/lib/python3.6/site-packages/bbopt/backends/hyperopt.py", line 137, in init
(next)(FMinIter(algo, domain, trials, rstate, **kwargs))
File "/home/b7066789/.local/lib/python3.6/site-packages/hyperopt/fmin.py", line 237, in next
self.run(1, block_until_done=self.asynchronous)
File "/home/b7066789/.local/lib/python3.6/site-packages/hyperopt/fmin.py", line 202, in run
self.rstate.randint(2 ** 31 - 1))
File "/home/b7066789/.local/lib/python3.6/site-packages/hyperopt/tpe.py", line 901, in suggest
print_node_on_error=False)
File "/home/b7066789/.local/lib/python3.6/site-packages/hyperopt/pyll/base.py", line 913, in rec_eval
rval = scope._impls[node.name](*args, **kwargs)
File "/home/b7066789/.local/lib/python3.6/site-packages/hyperopt/pyll/base.py", line 1076, in bincount
return np.bincount(x, weights, minlength)
TypeError: Cannot cast array data from dtype('<U4') to dtype('int64') according to the rule 'safe'

Add a framework for dealing with converting old bbopt data to new bbopt data

In case of the introduction of breaking changes to the data format, should have the ability to automatically convert data forward and tests to check that it works.

Not able to run the optimizer more than 10 times for Keras implementation

Hi!

I don't know if I should post this here but I encountered some difficulties trying to run your code on a Keras example more than 10 times at the 11th time I get an error like this:

TypeError: get_params() missing 1 required positional argument: 'self'

I don't why I get this if the code was able to run 10 times previously!
I hope you can help.

Thanks in advance :)

Add meta backend

Should work like the mixture backend except it uses a bandit strategy (eps greedy, boltzmann, boltzmann-gumbel) to pick between backends rather than a fixed mixture distribution.

Add pySOT backend

https://github.com/dme65/pySOT

run_backend with Mixture option

Hello, first of all very nice and easy to use package.
For some time on google colab i am trying to use the mixture functionality with multiple algorithms
but it always picks up "random" .
some code which i have used

jupyter cell1

from bbopt.backends.mixture import MixtureBackend
from bbopt import BlackBoxOptimizer

jupyter cell 2

bb = BlackBoxOptimizer("file", tag='index_'+str(comp_idx))

jupyter cell 3

bb.run_backend("mixture", [("random", 1), ('gaussian_process',1), ('random_forest',1), ('gradient_boosted_regression_trees',1)])
OR
bb.run_backend("mixture", [('gradient_boosted_regression_trees',1), ("random", 1), ('gaussian_process',1), ('random_forest',1) ])
......... Model code .....
if isinstance(bb.backend, MixtureBackend):
bb.remember({
"alg": bb.backend.selected_alg,
"l1_reg": l1_reg,
...... })
bb.maximize(vald_f1)

jupyter cell 4

loop = 70
opt_verbose = 0
for i in range(loop):
try:
run_me_RNN(opt_verbose)
print("Summary of run {}/{}:".format(i+1, loop))
pprint(bb.get_current_run())
print()
except Exception as e:
print('Something went wrong...', e)
print("Summary of run {}/{}:".format(i+1, loop))
pprint(bb.get_current_run())
print()

print("\nSummary of best run:")
pprint(bb.get_optimal_run())
####################################

I have tired a couple of times running the code but it always picks up "random" algorithm.
Do i need more iterations or is there something missing at my end.
On the other hand
#bb.run(alg='gaussian_process') #gradient_boosted_regression_trees
runs perfectly fine
#####################
On a side note , I am doing multi-label classification(unbalanced dataset), and taking validation-f1 score in maximize optimization. Just want to make sure if its the right error metrics that I am optimizing.
Some other parameters I have are training & validation loss , training f1-score.

Any help on the above issues will be greatly appreciated.

Many thanks

bbopt seems to be stuck on old version of networkx

I started playing with mycroft-precise for some personal stuff and bbopt seems stuck on a really old version of networkx.

That causes problems like this:
MycroftAI/mycroft-precise#92

and makes for ugly workarounds of nailing down versions like this:
MycroftAI/mycroft-precise#93

Compatibility with Colab?

Is Bbopt compatible with Jupyter Notebooks? I'm trying to test it in Colab, but I'm having issues getting it to work outside of a .py format.

Thanks for the help and the very cool package.

Support all scikit-optimize visualization methods

See skopt.plots. Currently, BBopt only supports plot_convergence. The idea would be to use BBopt's internal machinery for converting to skopt objects in plotting.

Add bbopt endpoint which runs many iterations

And get rid of the recommendation to use python file.py.