Hyperparameter optimization that enables researchers to experiment, visualize, and scale quickly.
Home Page: http://parameter-sherpa.readthedocs.io/
License: GNU General Public License v3.0
Hey guys,
I'm running mnist_mlp.ipynb from examples/ and in my case, at certain point, the algorithm repeats hyperparameter combinations:
Trial 6: {'num_units': 111, 'activation': 'relu'}
Trial 7: {'num_units': 111, 'activation': 'relu'}
Trial 10: {'num_units': 111, 'activation': 'relu'}
Trial 14: {'num_units': 111, 'activation': 'relu'}
I don't understand how could this be useful. I mean, shouldn't it keep track and avoid to reevaluate combinations?
After spamming {'num_units': 111, 'activation': 'relu'} (which is the best one found until now), it converges at non optimal configurations
Trial 26: {'num_units': 116, 'activation': 'sigmoid'}
after many trials of similar configurations...
Trial 72: {'num_units': 115, 'activation': 'sigmoid'}
I know you you are just wrapping this algorithm from another repo, but, as I'm getting this running one of your examples, I guess you have probably faced this questions as well
thanks
in https://parameter-sherpa.readthedocs.io/en/latest/gettingstarted/guide.html
I think the second line should be
model = init_model(trial.parameters)
I might be wrong, thanks for the tool!
The scheduler splits the command with spaces: https://github.com/LarsHH/sherpa/blob/master/sherpa/schedulers.py#L110
This means any file name / file path with space will not work. This should either be changed or mentioned in the docs.
I am running sherpa with tensorflow 2.1 on google colab. The first problem I ran into was that the previous link explaining how to run sherpa with tensorflow 2.1 in google didn't work fortunately I was able to resolve this and pull up the dashboard using
from tensorboard import notebook
notebook.display(port=8880, height=1000)
Unfortunately it is not printing anything to the dashboard. I believe that it may be related to this error which I am receiving in my final line of code after it finishes running all the trials
---> 34 study.finalize(trial=trial)
TypeError: reduction operation 'argmin' not allowed for this dtype
Simply.py fails in two ways.
def main():
...
if __name__ == '__main__':
main()This causes any new process to crash as the file parsing has not completed.
$ python simple.py
INFO:sherpa.core:
-------------------------------------------------------
SHERPA Dashboard running on http://192.168.1.67:8999
-------------------------------------------------------
Traceback (most recent call last):
File "simple.py", line 73, in <module>
main()
File "simple.py", line 40, in main
dashboard_port=8999)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36\lib\site-packages\sherpa\core.py", line 98, in __init__
self.dashboard_process = self._run_web_server(dashboard_port)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36\lib\site-packages\sherpa\core.py", line 299, in _run_web_server
proc.start()
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 105, in start
self._popen = self._Popen(self)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
reduction.dump(process_obj, to_child)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects
Ian@BIANCA-TOWER C:\Users\Ian\clones\sherpa\examples
$ Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_main
exitcode = _main(fd)
File "C:\Users\Ian\AppData\Local\Programs\Python\Python36\lib\multiprocessing\spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input
First things first, great work on the library! 👍
This (very minor) issue is regarding this line, where I believe shlex.split(command) would be more appropriate (instead of command.split(' ') (shlex is a builtin module). It would enable commands like --arg1 "Argument 1" --arg2 "Argument 2". It would also be nice if command could be a pre-parsed list.
Since MongoDB does not work correctly (#64) I have rewritten the mnistcnnpbt example (https://github.com/sherpa-ai/sherpa/tree/master/examples/parallel-examples/mnistcnnpbt) as follows:
(I have significantly reduced the training and test data set, so that the training on the CPU is performed faster)
https://github.com/martsalz/sherpa-PBT/blob/master/MNIST%20PBT%20Without%20MongoDB.ipynb
I expected through the following line in the code the learning rate is changed from generation to generation - but this is not the case: K.set_value(model.optimizer.lr, trial.parameters['lr'])
As I understand the code, the following generations will load the already trained models and modify their parameters (the new modified parameters will be provided by the sherpa.algorithms.PopulationBasedTraining). How should the code be changed, for example, to change the batch_size or the dropout_rate in one generation?
On laptop the link only works if it's something like localhost:8888, on cluster this only works if its xxx.xxx.xxx.xxx:8888. Is there a uniform way to print the appropriate address?
Hi,
When I run my code, the links provided to access the dashboard are invalid. I tried using the key disable_dashboard = True in sherpa.Study and then after running all my code, sherpa.Study.load_dashboard(".") to load the dashboard, but those links do not work either.
Here is the code:
parameters = [sherpa.Continuous(name='lr', range=[0.005, 0.1], scale='log'),
sherpa.Continuous(name='dropout', range=[0., 0.4]),
sherpa.Ordinal(name='batch_size', range=[16, 32, 64, 256, 512, 1024]),
sherpa.Discrete(name='num_hidden_units', range=[1, 300]),
sherpa.Choice(name='activation', range=['sigmoid', 'relu'])]
algorithm = sherpa.algorithms.RandomSearch(max_num_trials=10)
study = sherpa.Study(parameters=parameters,
algorithm=algorithm, disable_dashboard=False,
lower_is_better=False)
for trial in study:
model = Sequential()
model.add(Dense(units=trial.parameters['num_hidden_units'], activation=trial.parameters['activation'], input_dim=np.shape(Xin)[1]))
model.add(Dense(np.shape(Yout)[1], activation='softmax'))
model.compile(loss='binary_crossentropy', optimizer=optimizers.Adam(trial.parameters['lr']), metrics=['accuracy'])
model.fit(x_train, y_train, epochs=5, batch_size=trial.parameters['batch_size'])
loss,acc = model.evaluate(x_test,y_test)
study.add_observation(trial=trial, objective=acc, context={'loss':loss})
if study.should_trial_stop(trial):
break
study.finalize(trial)
study.save(".")
I ran sherpa code in my jupyter notebook on laptop(Windows) which connects to a server(Ubuntu) and Sherpa study gave following link for dashboard: http://172.17.0.2:8880.
On server, there is docker running which was started with below command:
nvidia-docker run -it --rm -v /home/user/workdir/:/root/mywork -p 8888:8888 -p 6006:6006 -p 8880:8880 tensorflow/tensorflow:latest-gpu-py3 "/bin/bash"
-p 8888:8888: used for Tensorflow
-p 6006:6006 : used for Tensorboard
(I tried both with and without -p 8880:8880 option)
For tunneling I ran below command:
plink -ssh -T -L 8880:server:8880 user@server
But nothing gets displayed when I try to open the dashboard link, it shows connection timed out.
It would be useful to be able to pass custom parameters to mongodb (such as the location of a config file).
In my particular case, I had to install mongodb in my user space since I don't have admin privileges, so I have to point it to a custom location for the config file.
Dear Lars
I experience a KeyError after random number of trials with RandomSearch and ASHA. It was related to my pandas version > 1.1.0. Editing Line 189 in core.py to capture KeyError and not TypeError fixed the problem.
the release Notes of pandas 1.1.0 states:
"Label lookups series[key], series.loc[key] and frame.loc[key] used to raise either KeyError or TypeError depending on the type of key and type of Index. These now consistently raise KeyError (GH31867)"
Thanks
I have installed all required packages and executed the above example. The web interface is not reachable. An error message etc. is not displayed - what could be the reason?
Running in a venv under Python 3.5
The parameter sherpa.Discrete('num_units', [32, 128]) can produce floats after some trials.
To reproduce, install sherpa (!pip3 install parameter-sherpa gpyopt flask) and run the mnist_mlp.ipynb example on Google Colab.
Error output:
INFO:GP:initializing Y
INFO:GP:initializing inference method
INFO:GP:adding kernel and likelihood as parameters
WARNING:rbf:reconstraining parameters GP_regression.rbf
WARNING:variance:reconstraining parameters GP_regression.Gaussian_noise.variance
Trial 4: {'num_units': 125.0, 'activation': 'tanh'}
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-13-d3ff5f70567f> in <module>()
5
6 model = Sequential()
----> 7 model.add(Dense(units=trial.parameters['num_units'], activation=trial.parameters['activation'], input_dim=784))
8 model.add(Dense(units=10, activation='softmax'))
9 model.compile(loss='categorical_crossentropy',
10 frames
/usr/local/lib/python3.6/dist-packages/keras/engine/sequential.py in add(self, layer)
163 # and create the node connecting the current layer
164 # to the input layer we just created.
--> 165 layer(x)
166 set_inputs = True
167 else:
/usr/local/lib/python3.6/dist-packages/keras/engine/base_layer.py in __call__(self, inputs, **kwargs)
429 'You can build it manually via: '
430 '`layer.build(batch_input_shape)`')
--> 431 self.build(unpack_singleton(input_shapes))
432 self.built = True
433
/usr/local/lib/python3.6/dist-packages/keras/layers/core.py in build(self, input_shape)
864 name='kernel',
865 regularizer=self.kernel_regularizer,
--> 866 constraint=self.kernel_constraint)
867 if self.use_bias:
868 self.bias = self.add_weight(shape=(self.units,),
/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
89 warnings.warn('Update your `' + object_name + '` call to the ' +
90 'Keras 2 API: ' + signature, stacklevel=2)
---> 91 return func(*args, **kwargs)
92 wrapper._original_function = func
93 return wrapper
/usr/local/lib/python3.6/dist-packages/keras/engine/base_layer.py in add_weight(self, name, shape, dtype, initializer, regularizer, trainable, constraint)
247 if dtype is None:
248 dtype = K.floatx()
--> 249 weight = K.variable(initializer(shape),
250 dtype=dtype,
251 name=name,
/usr/local/lib/python3.6/dist-packages/keras/initializers.py in __call__(self, shape, dtype)
216 limit = np.sqrt(3. * scale)
217 return K.random_uniform(shape, -limit, limit,
--> 218 dtype=dtype, seed=self.seed)
219
220 def get_config(self):
/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py in random_uniform(shape, minval, maxval, dtype, seed)
4137 seed = np.random.randint(10e6)
4138 return tf.random_uniform(shape, minval=minval, maxval=maxval,
-> 4139 dtype=dtype, seed=seed)
4140
4141
/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/random_ops.py in random_uniform(shape, minval, maxval, dtype, seed, name)
245 shape, minval, maxval, seed=seed1, seed2=seed2, name=name)
246 else:
--> 247 rnd = gen_random_ops.random_uniform(shape, dtype, seed=seed1, seed2=seed2)
248 return math_ops.add(rnd * (maxval - minval), minval, name=name)
249
/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_random_ops.py in random_uniform(shape, dtype, seed, seed2, name)
775 _, _, _op = _op_def_lib._apply_op_helper(
776 "RandomUniform", shape=shape, dtype=dtype, seed=seed, seed2=seed2,
--> 777 name=name)
778 _result = _op.outputs[:]
779 _inputs_flat = _op.inputs
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py in _apply_op_helper(self, op_type_name, name, **keywords)
608 _SatisfiesTypeConstraint(base_type,
609 _Attr(op_def, input_arg.type_attr),
--> 610 param_name=input_name)
611 attrs[input_arg.type_attr] = attr_value
612 inferred_from[input_arg.type_attr] = input_name
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py in _SatisfiesTypeConstraint(dtype, attr_def, param_name)
58 "allowed values: %s" %
59 (param_name, dtypes.as_dtype(dtype).name,
---> 60 ", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
61
62
TypeError: Value passed to parameter 'shape' has DataType float32 not in list of allowed values: int32, int64
Hi,
Is there a way to resume a saved sherpa run? I noticed there was a load_dashbord() method, but it also says "currently this study cannot be used to continue the optimization".
Moreover, I was wondering if it was possible to start with a given algorithm (e.g. random search), save the study, and resume it with another one (e.g. Bayesian optimization).
Thank you!
I'm running into issues trying to run the dashboard on a remote cluster.
Running the statement
study = sherpa.Study(parameters=parameters, algorithm=alg, lower_is_better=True)
results in the following:
Process Process-2: Traceback (most recent call last): File "/hpcdata/usafa9/config/python/intelpython3/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/hpcdata/usafa9/config/python/intelpython3/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/hpcdata/usafa9/config/python/intelpython3/lib/python3.6/site-packages/flask/app.py", line 938, in run cli.show_server_banner(self.env, self.debug, self.name, False) File "/hpcdata/usafa9/config/python/intelpython3/lib/python3.6/site-packages/flask/cli.py", line 629, in show_server_banner click.echo(message) File "/hpcdata/usafa9/config/python/intelpython3/lib/python3.6/site-packages/click/utils.py", line 262, in echo file.write(message) io.UnsupportedOperation: not writable
The issue stems from the Click module, around line 218 of utils.py, which is file = _default_text_stdout(). Replacing this with file = sys.stdout resolves the issue for me.
I guess this is more an issue with Click than with Sherpa, but since I think my use case (running Sherpa on a remote cluster and accessing the dashboard via ssh tunneling) is probably common, I wanted to make you aware of it.
Running the example mnist_mlp, GPyOpt gave me the following configuration:
Trial 4: {'num_units': 116.0, 'activation': 'relu'}
This obviously makes Keras to produce an error, as it does not admit real values for number of units
I know this is probably due to the changes made to solve the issue #42
Invalid syntax in hyperparameters:
Right now in docs:
parameters = [sherpa.Continuous(name='lr', range=[0.005, 0.1], scale='log'),
sherpa.Continuous(name='dropout', range=[0., 0.4]),
sherpa.Ordinal(name='batch_size', range=[16, 32, 64]),
sherpa.Discrete(name='num_hidden_units', range=[100, 300]),
sherpa.Choice(name'activation', range=['relu', 'elu', 'prelu'])]
Should be:
parameters = [sherpa.Continuous(name='lr', range=[0.005, 0.1], scale='log'),
sherpa.Continuous(name='dropout', range=[0., 0.4]),
sherpa.Ordinal(name='batch_size', range=[16, 32, 64]),
sherpa.Discrete(name='num_hidden_units', range=[100, 300]),
sherpa.Choice(name='activation', range=['relu', 'elu', 'prelu'])]
When running in parallel and trial crashes the displayed location of the logs is automatically output_dir + /sge/ + "trial_x.out". This is even the case if running with LocalScheduler and there won't be any logs in that location.
Initialization of the keras models between trials.
That sometimes has a large effect on results. I think a random seed is worth mentioning in the tutorial.
It seems that a maximum of 28 columns are displayed in the dashboard grid, truncating remaining values and the objective value. These values are displayed in other parts of the dashboard, in results csvs, etc. Upon googling, it seems like this is an issue with older versions of slickgrid.
Apologies for the inappropriate issue submission, but is there a known JavaScript port of this library?
When using the bayesian optimization algorithm, I get the following error:
ValueError: `f0` passed has more than 1 dimension.
An extract from the output is
Creating new model for trial 6...
INFO:GP:initializing Y
INFO:GP:initializing inference method
INFO:GP:adding kernel and likelihood as parameters
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-9-bd36a65e8a36> in <module>
----> 1 run_study()
<ipython-input-8-768b946e96b2> in run_study()
15 dashboard_port=8877)
16 steps = 3
---> 17 for trial in study:
18 print("-"*100)
19 print(f"Trial:\t{trial.id}\nSteps:\t{steps}\nParameters:{trial.parameters}\n")
~/host/src/sherpa/sherpa/core.py in __next__(self)
377 Allows to write `for trial in study:`.
378 """
--> 379 t = self.get_suggestion()
380 if isinstance(t, Trial):
381 return t
~/host/src/sherpa/sherpa/core.py in get_suggestion(self)
214
215 p = self.algorithm.get_suggestion(self.parameters, self.results,
--> 216 self.lower_is_better)
217 if isinstance(p, dict):
218 self.num_trials += 1
~/host/src/sherpa/sherpa/algorithms/bayesian_optimization.py in get_suggestion(self, parameters, results, lower_is_better)
109
110 domain = self._initialize_domain(parameters)
--> 111 batch = self._generate_bayesopt_batch(X, y, lower_is_better, domain)
112
113 batch_list_of_dicts = self._reverse_to_sherpa_format(batch,
~/host/src/sherpa/sherpa/algorithms/bayesian_optimization.py in _generate_bayesopt_batch(self, X, y, lower_is_better, domain)
142 exact_feval=False,
143 model_type=self.model_type)
--> 144 return bo_step.suggest_next_locations()
145
146 def get_best_pred(self, parameters, results, lower_is_better):
/usr/local/python/lib/python3.6/site-packages/GPyOpt/core/bo.py in suggest_next_locations(self, context, pending_X, ignored_X)
67 self._update_model(self.normalization_type)
68
---> 69 suggested_locations = self._compute_next_evaluations(pending_zipped_X = pending_X, ignored_zipped_X = ignored_X)
70
71 return suggested_locations
/usr/local/python/lib/python3.6/site-packages/GPyOpt/core/bo.py in _compute_next_evaluations(self, pending_zipped_X, ignored_zipped_X)
234
235 ### We zip the value in case there are categorical variables
--> 236 return self.space.zip_inputs(self.evaluator.compute_batch(duplicate_manager=duplicate_manager, context_manager= self.acquisition.optimizer.context_manager))
237
238 def _update_model(self, normalization_type='stats'):
/usr/local/python/lib/python3.6/site-packages/GPyOpt/core/evaluators/batch_local_penalization.py in compute_batch(self, duplicate_manager, context_manager)
35 if self.batch_size >1:
36 # ---------- Approximate the constants of the the method
---> 37 L = estimate_L(self.acquisition.model.model,self.acquisition.space.get_bounds())
38 Min = self.acquisition.model.model.Y.min()
39
/usr/local/python/lib/python3.6/site-packages/GPyOpt/core/evaluators/batch_local_penalization.py in estimate_L(model, bounds, storehistory)
64 pred_samples = df(samples,model,0)
65 x0 = samples[np.argmin(pred_samples)]
---> 66 res = scipy.optimize.minimize(df,x0, method='L-BFGS-B',bounds=bounds, args = (model,x0), options = {'maxiter': 200})
67 minusL = res.fun[0][0]
68 L = -minusL
/usr/local/python/lib/python3.6/site-packages/scipy/optimize/_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
616 elif meth == 'l-bfgs-b':
617 return _minimize_lbfgsb(fun, x0, args, jac, bounds,
--> 618 callback=callback, **options)
619 elif meth == 'tnc':
620 return _minimize_tnc(fun, x0, args, jac, bounds, callback=callback,
/usr/local/python/lib/python3.6/site-packages/scipy/optimize/lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, finite_diff_rel_step, **unknown_options)
306 sf = _prepare_scalar_function(fun, x0, jac=jac, args=args, epsilon=eps,
307 bounds=new_bounds,
--> 308 finite_diff_rel_step=finite_diff_rel_step)
309
310 func_and_grad = sf.fun_and_grad
/usr/local/python/lib/python3.6/site-packages/scipy/optimize/optimize.py in _prepare_scalar_function(fun, x0, jac, args, bounds, epsilon, finite_diff_rel_step, hess)
260 # calculation reduces overall function evaluations.
261 sf = ScalarFunction(fun, x0, args, grad, hess,
--> 262 finite_diff_rel_step, bounds, epsilon=epsilon)
263
264 return sf
/usr/local/python/lib/python3.6/site-packages/scipy/optimize/_differentiable_functions.py in __init__(self, fun, x0, args, grad, hess, finite_diff_rel_step, finite_diff_bounds, epsilon)
93
94 self._update_grad_impl = update_grad
---> 95 self._update_grad()
96
97 # Hessian Evaluation
/usr/local/python/lib/python3.6/site-packages/scipy/optimize/_differentiable_functions.py in _update_grad(self)
169 def _update_grad(self):
170 if not self.g_updated:
--> 171 self._update_grad_impl()
172 self.g_updated = True
173
/usr/local/python/lib/python3.6/site-packages/scipy/optimize/_differentiable_functions.py in update_grad()
90 self.ngev += 1
91 self.g = approx_derivative(fun_wrapped, self.x, f0=self.f,
---> 92 **finite_diff_options)
93
94 self._update_grad_impl = update_grad
/usr/local/python/lib/python3.6/site-packages/scipy/optimize/_numdiff.py in approx_derivative(fun, x0, method, rel_step, abs_step, f0, bounds, sparsity, as_linear_operator, args, kwargs)
386 f0 = np.atleast_1d(f0)
387 if f0.ndim > 1:
--> 388 raise ValueError("`f0` passed has more than 1 dimension.")
389
390 if np.any((x0 < lb) | (x0 > ub)):
ValueError: `f0` passed has more than 1 dimension.
The relevant portion of my code is as follows
def run_study():
parameters = [
sherpa.Continuous('learning_rate', [1e-6, 1e-1], 'log'),
sherpa.Choice('activation', ['relu', 'tanh', 'logistic']),
sherpa.Continuous('alpha', [1e-7, 0.1], 'log'),
sherpa.Discrete('no_layers', [1, 6]),
sherpa.Continuous('fan_out', [2.0, 4.0])
]
algorithm = sherpa.algorithms.GPyOpt(max_num_trials=150)
study = sherpa.Study(parameters=parameters,
algorithm=algorithm,
lower_is_better=False,
dashboard_port=8877)
steps = 3
for trial in study:
print("-"*100)
print(f"Trial:\t{trial.id}\nSteps:\t{steps}\nParameters:{trial.parameters}\n")
print(f"Creating new model for trial {trial.id}...\n")
# Get hyperparameters
lr = trial.parameters['learning_rate']
act = trial.parameters['activation']
alpha = trial.parameters['alpha']
nl = trial.parameters['no_layers']
f_out = trial.parameters['fan_out']
hidden_layers = get_hidden_layers(nl, f_out)
res = run_trial(data, steps, lr, 1000, alpha, act, hidden_layers)
study.add_observation(trial=trial, iteration=1, objective=res)
study.finalize(trial=trial)
Sherpa commit: 4ad600c71f76a06993464e5078188f552111b851
Python version: 3.6.3
scipy version: 1.5.0
GPyOpt Version: 1.2.6
Is this a bug or am I doing something wrong?
I'm trying to implement sherpa for parallel bayesian optimization, but just trying to run the sherpa/examples/parallel_example/simple.py throws the following errors:
"""
SHERPA is a Python library for hyperparameter tuning of machine learning models.
Copyright (C) 2018 Lars Hertel, Peter Sadowski, and Julian Collado.
This file is part of SHERPA.
SHERPA is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
SHERPA is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with SHERPA. If not, see <http://www.gnu.org/licenses/>.
"""
from __future__ import print_function
import tempfile
import os
import sherpa
import sherpa.schedulers
import argparse
import socket
import sherpa.algorithms.bayesian_optimization as bayesian_optimization
parser = argparse.ArgumentParser()
parser.add_argument('--env', help='Your environment path.',
default='/home/lhertel/profiles/python3env.profile', type=str)
FLAGS = parser.parse_args()
# figuring out host and queue
host = socket.gethostname()
sge_q = 'arcus.q' if (host.startswith('arcus-1') or host.startswith('arcus-2') or host.startswith('arcus-3') or host.startswith('arcus-4')) else 'arcus-ubuntu.q'
tempdir = tempfile.mkdtemp(dir=".")
parameters = [sherpa.Choice(name="param_a",
range=[1, 2, 3]),
sherpa.Continuous(name="param_b",
range=[0, 1])]
algorithm = sherpa.algorithms.RandomSearch(max_num_trials=10)
# stopping_rule = sherpa.algorithms.MedianStoppingRule(min_iterations=2,
# min_trials=3)
# algorithm = bayesian_optimization.GPyOpt(max_concurrent=4,
# model_type='GP',
# acquisition_type='EI',
# max_num_trials=100)
# scheduler = sherpa.schedulers.SGEScheduler(submit_options="-N example -P arcus.p -q {} -l hostname='{}'".format(sge_q, host), environment=FLAGS.env, output_dir=tempdir)
scheduler = sherpa.schedulers.LocalScheduler()
### The *training script*
testscript = """import sherpa
import time
client = sherpa.Client(db_dir = "mongodb://localhost:27017/sherpa_sample/jobs/", port = 27017)
trial = client.get_trial()
# Simulate model training
num_iterations = 10
for i in range(num_iterations):
pseudo_objective = trial.parameters['param_a'] / float(i + 1) * trial.parameters['param_b']
time.sleep(1)
client.send_metrics(trial=trial, iteration=i+1,
objective=pseudo_objective)
# print("Trial {} Iteration {}.".format(trial.id, i+1))
# print("Trial {} finished.".format(trial.id))
"""
dbport =27017
filename = os.path.join(tempdir, "test.py")
with open(filename, 'w') as f:
f.write(testscript)
results = sherpa.optimize(parameters=parameters,
algorithm=algorithm,
lower_is_better=True,
filename=filename,
output_dir=tempdir,
scheduler=scheduler,
max_concurrent=4,
verbose=1)
print(results)
INFO:sherpa.core:
* Serving Flask app "sherpa.app.app" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: on
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 1:
param_a = 2
param_b = 0.8522431562647138
-------------------------------------------------------
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 2:
param_a = 3
param_b = 0.29715833725982344
-------------------------------------------------------
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 3:
param_a = 2
param_b = 0.020480942550347048
-------------------------------------------------------
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 4:
param_a = 3
param_b = 0.596307236230985
-------------------------------------------------------
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 5:
param_a = 1
param_b = 0.4686961645249037
-------------------------------------------------------
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 6:
param_a = 2
param_b = 0.35682090172629666
-------------------------------------------------------
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 7:
param_a = 2
param_b = 0.6391827285407945
-------------------------------------------------------
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 8:
param_a = 3
param_b = 0.4170850070420954
-------------------------------------------------------
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 9:
param_a = 1
param_b = 0.250596506613348
-------------------------------------------------------
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 10:
param_a = 1
param_b = 0.4596871330102428
-------------------------------------------------------
INFO:sherpa.core:Optimization Algorithm finished.
INFO:sherpa.core:Optimization Algorithm finished.
Closing MongoDB!
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2657, in get_loc
return self._engine.get_loc(key)
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 1601, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1608, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Objective'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "simple.py", line 87, in <module>
verbose=1)
File "/usr/local/lib/python3.6/dist-packages/sherpa/core.py", line 636, in optimize
return study.get_best_result()
File "/usr/local/lib/python3.6/dist-packages/sherpa/core.py", line 254, in get_best_result
self.lower_is_better)
File "/usr/local/lib/python3.6/dist-packages/sherpa/algorithms/core.py", line 69, in get_best_result
if lower_is_better
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 1494, in __getitem__
return self._getitem_tuple(key)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 868, in _getitem_tuple
return self._getitem_lowerdim(tup)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 988, in _getitem_lowerdim
section = self._getitem_axis(key, axis=i)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 1913, in _getitem_axis
return self._get_label(key, axis=axis)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 141, in _get_label
return self.obj._xs(label, axis=axis)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/generic.py", line 3576, in xs
return self[key]
File "/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py", line 2927, in __getitem__
indexer = self.columns.get_loc(key)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2659, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 1601, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1608, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Objective'
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
import apport.fileutils
File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
from apport.packaging_impl import impl as packaging
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 23, in <module>
import apt
File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'
Original exception was:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2657, in get_loc
return self._engine.get_loc(key)
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 1601, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1608, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Objective'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "simple.py", line 87, in <module>
verbose=1)
File "/usr/local/lib/python3.6/dist-packages/sherpa/core.py", line 636, in optimize
return study.get_best_result()
File "/usr/local/lib/python3.6/dist-packages/sherpa/core.py", line 254, in get_best_result
self.lower_is_better)
File "/usr/local/lib/python3.6/dist-packages/sherpa/algorithms/core.py", line 69, in get_best_result
if lower_is_better
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 1494, in __getitem__
return self._getitem_tuple(key)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 868, in _getitem_tuple
return self._getitem_lowerdim(tup)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 988, in _getitem_lowerdim
section = self._getitem_axis(key, axis=i)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 1913, in _getitem_axis
return self._get_label(key, axis=axis)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexing.py", line 141, in _get_label
return self.obj._xs(label, axis=axis)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/generic.py", line 3576, in xs
return self[key]
File "/usr/local/lib/python3.6/dist-packages/pandas/core/frame.py", line 2927, in __getitem__
indexer = self.columns.get_loc(key)
File "/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py", line 2659, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx", line 108, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 1601, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1608, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Objective'
Could you please explain the exact difference between parameters and parameter_range?
Why are the parameters specified twice?
https://parameter-sherpa.readthedocs.io/en/latest/algorithms/pbt.html
Thanks.
AttributeError: module 'sherpa.algorithms' has no attribute 'BayesianOptimization'
I am already install the gpyopt, but still got this error.
I have tried running the parallel simple.py and mnistmlp examples, but when I do, I keep getting the following error in the jobs/trial_*.out files about connecting to the database.
2020-09-03 10:40:27.957058: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
warning in stationary: failed to import cython module: falling back to numpy
warning in coregionalize: failed to import cython module: falling back to numpy
warning in choleskies: failed to import cython module: falling back to numpy
Traceback (most recent call last):
File "trial.py", line 79, in <module>
trial = client.get_trial()
File "/glade/u/home/dgagne/miniconda3/envs/goes/lib/python3.7/site-packages/sherpa/database.py", line 222, in get_trial
t = next(g)
File "/glade/u/home/dgagne/miniconda3/envs/goes/lib/python3.7/site-packages/sherpa/database.py", line 221, in <genexpr>
g = (entry for entry in self.db.trials.find({'trial_id': trial_id}))
File "/glade/u/home/dgagne/miniconda3/envs/goes/lib/python3.7/site-packages/pymongo/cursor.py", line 1207, in next
if len(self.__data) or self._refresh():
File "/glade/u/home/dgagne/miniconda3/envs/goes/lib/python3.7/site-packages/pymongo/cursor.py", line 1100, in _refresh
self.__session = self.__collection.database.client._ensure_session()
File "/glade/u/home/dgagne/miniconda3/envs/goes/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1816, in _ensure_session
return self.__start_session(True, causal_consistency=False)
File "/glade/u/home/dgagne/miniconda3/envs/goes/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1766, in __start_session
server_session = self._get_server_session()
File "/glade/u/home/dgagne/miniconda3/envs/goes/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1802, in _get_server_session
return self._topology.get_server_session()
File "/glade/u/home/dgagne/miniconda3/envs/goes/lib/python3.7/site-packages/pymongo/topology.py", line 488, in get_server_session
None)
File "/glade/u/home/dgagne/miniconda3/envs/goes/lib/python3.7/site-packages/pymongo/topology.py", line 217, in _select_servers_loop
(self._error_message(selector), timeout, self.description))
pymongo.errors.ServerSelectionTimeoutError: casper26:27001: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 5f511c8580fc9c3448b850b1, topology_type: Single, servers: [<ServerDescription ('casper26', 27001) server_type: Unknown, rtt: None, error=AutoReconnect('casper26:27001: [Errno 111] Connection refused')>]>
Any ideas on what may be going wrong? I installed mongodb through conda. The main program also completes with no errors, but there are no summary results at the end.
Currently flask is running a development server. Use production server instead:
* Environment: production
WARNING: Do not use the development server in a production environment.
Use a production WSGI server instead.
* Debug mode: on
Suppose that 2 different trials are stopped, one of them has a terribly bad objective score, the other one has a decent objective score (just a bit worse than the stopping threshold). Does Sherpa take those scores into account when sampling new parameters? In other words, does it use the information obtained frome pruned (i.e., early stopped) trials in the optimization process or is that information discarded/ignored? I ask because considering that information may avoid sampling (i.e., suggesting) unpromising hyperparameters (that were pruned) multiple times.
MongoDB does not accept Numpy float 32 and others. Make sure that what gets submitted is converted properly or a warning/error is thrown.
When I run the mnistcnnpbt example, the following error message appears:
(Always at trial 20)
INFO:sherpa.core:
-------------------------------------------------------
Submitting Trial 20:
dropout = 0.30687144848463904
load_from = 6
lr = 0.007697357252133577
save_to = 20
lineage = 6,
batch_size = 16
generation = 2
-------------------------------------------------------
Closing MongoDB!
Traceback (most recent call last):
File "runner.py", line 67, in <module>
output_dir='./output')
File "/home/martin/Desktop/sherpa/sherpa/core.py", line 640, in optimize
runner.run_loop()
File "/home/martin/Desktop/sherpa/sherpa/core.py", line 561, in run_loop
self.submit_new_trials()
File "/home/martin/Desktop/sherpa/sherpa/core.py", line 522, in submit_new_trials
next_trial = self.study.get_suggestion()
File "/home/martin/Desktop/sherpa/sherpa/core.py", line 200, in get_suggestion
self.lower_is_better)
File "/home/martin/Desktop/sherpa/sherpa/algorithms/core.py", line 576, in get_suggestion
lower_is_better=lower_is_better)
File "/home/martin/Desktop/sherpa/sherpa/algorithms/core.py", line 601, in _truncation_selection
d = generation_df.iloc[(self.count - 1) % self.population_size].to_dict()
File "/home/martin/Desktop/venv_sherpa/lib/python3.5/site-packages/pandas/core/indexing.py", line 1328, in __getitem__
return self._getitem_axis(key, axis=0)
File "/home/martin/Desktop/venv_sherpa/lib/python3.5/site-packages/pandas/core/indexing.py", line 1749, in _getitem_axis
self._is_valid_integer(key, axis)
File "/home/martin/Desktop/venv_sherpa/lib/python3.5/site-packages/pandas/core/indexing.py", line 1638, in _is_valid_integer
raise IndexError("single positional indexer is out-of-bounds")
IndexError: single positional indexer is out-of-bounds
(venv_sherpa) martin@martin-notebook:~/Desktop/sherpa/examples/parallel-examples/mnistcnnpbt$
When I run this on Linux command line from an activated conda environment:
pip install parameter-sherpaI get this error:
Looking in indexes: https://[PROXY-TO-PYPI]/pypi/pypi-releases/simple
Collecting parameter-sherpa
Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/e4/6a/96dac197a9c8f3aa82a83a2805bc40ba615613e35a715e70912385670fb7/parameter-sherpa-1.0.5.tar.gz (513kB)
|████████████████████████████████| 522kB 600kB/s
ERROR: Command errored out with exit status 1:
command: /app/local/anaconda3/envs/my-conda-env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-spl_3eo8/parameter-sherpa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-spl_3eo8/parameter-sherpa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-spl_3eo8/parameter-sherpa/
Complete output (11 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-spl_3eo8/parameter-sherpa/setup.py", line 13, in <module>
import sherpa
File "/tmp/pip-install-spl_3eo8/parameter-sherpa/sherpa/__init__.py", line 21, in <module>
from .core import *
File "/tmp/pip-install-spl_3eo8/parameter-sherpa/sherpa/core.py", line 32, in <module>
from .database import _Database
File "/tmp/pip-install-spl_3eo8/parameter-sherpa/sherpa/database.py", line 22, in <module>
import pymongo
ModuleNotFoundError: No module named 'pymongo'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
So I then manually installed pymongo and tried again. I then received this error:
Looking in indexes: https://[PROXY-TO-PYPI]/pypi/pypi-releases/simple
Collecting parameter-sherpa
Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/e4/6a/96dac197a9c8f3aa82a83a2805bc40ba615613e35a715e70912385670fb7/parameter-sherpa-1.0.5.tar.gz (513kB)
|████████████████████████████████| 522kB 1.4MB/s
ERROR: Command errored out with exit status 1:
command: /app/local/anaconda3/envs/my-conda-env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-19323g_f/parameter-sherpa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-19323g_f/parameter-sherpa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-19323g_f/parameter-sherpa/
Complete output (11 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-19323g_f/parameter-sherpa/setup.py", line 13, in <module>
import sherpa
File "/tmp/pip-install-19323g_f/parameter-sherpa/sherpa/__init__.py", line 24, in <module>
from . import algorithms
File "/tmp/pip-install-19323g_f/parameter-sherpa/sherpa/algorithms/__init__.py", line 20, in <module>
from .core import *
File "/tmp/pip-install-19323g_f/parameter-sherpa/sherpa/algorithms/core.py", line 28, in <module>
import sklearn.gaussian_process
ModuleNotFoundError: No module named 'sklearn'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
Further, after I manually installed the scikit-learn package, I got another similar error, this time for GPyOpt.
Looking in indexes: https://[PROXY-TO-PYPI]/pypi/pypi-releases/simple
Collecting parameter-sherpa
Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/e4/6a/96dac197a9c8f3aa82a83a2805bc40ba615613e35a715e70912385670fb7/parameter-sherpa-1.0.5.tar.gz (513kB)
|████████████████████████████████| 522kB 1.5MB/s
ERROR: Command errored out with exit status 1:
command: /app/local/anaconda3/envs/my-conda-env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-pfmvbs6a/parameter-sherpa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-pfmvbs6a/parameter-sherpa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
cwd: /tmp/pip-install-pfmvbs6a/parameter-sherpa/
Complete output (11 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-pfmvbs6a/parameter-sherpa/setup.py", line 13, in <module>
import sherpa
File "/tmp/pip-install-pfmvbs6a/parameter-sherpa/sherpa/__init__.py", line 24, in <module>
from . import algorithms
File "/tmp/pip-install-pfmvbs6a/parameter-sherpa/sherpa/algorithms/__init__.py", line 22, in <module>
from .bayesian_optimization import GPyOpt
File "/tmp/pip-install-pfmvbs6a/parameter-sherpa/sherpa/algorithms/bayesian_optimization.py", line 8, in <module>
import GPyOpt as gpyopt_package
ModuleNotFoundError: No module named 'GPyOpt'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
So I then had to manually run pip install GPyOpt.
After that, I was then able to successfully run pip install parameter-sherpa. In this case, some other dependencies actually were successfully resolved:
Looking in indexes: https://[PROXY-TO-PYPI]/pypi/pypi-releases/simple
Collecting parameter-sherpa
Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/e4/6a/96dac197a9c8f3aa82a83a2805bc40ba615613e35a715e70912385670fb7/parameter-sherpa-1.0.5.tar.gz (513kB)
|████████████████████████████████| 522kB 751kB/s
Requirement already satisfied: pandas>=0.20.3 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from parameter-sherpa) (0.25.1)
Requirement already satisfied: pymongo>=3.5.1 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from parameter-sherpa) (3.9.0)
Requirement already satisfied: numpy>=1.8.2 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from parameter-sherpa) (1.16.5)
Requirement already satisfied: scipy>=1.0.0 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from parameter-sherpa) (1.3.1)
Requirement already satisfied: scikit-learn>=0.19.1 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from parameter-sherpa) (0.21.3)
Collecting flask>=0.12.2 (from parameter-sherpa)
Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl (94kB)
|████████████████████████████████| 102kB 989kB/s
Collecting drmaa>=0.7.7 (from parameter-sherpa)
Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/de/3f/3f74887d9ba45c34a9b12c754d064ba2a8f9458d6ab3967915d4337bc6df/drmaa-0.7.9.tar.gz
Requirement already satisfied: python-dateutil>=2.6.1 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from pandas>=0.20.3->parameter-sherpa) (2.8.0)
Requirement already satisfied: pytz>=2017.2 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from pandas>=0.20.3->parameter-sherpa) (2019.2)
Requirement already satisfied: joblib>=0.11 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from scikit-learn>=0.19.1->parameter-sherpa) (0.14.0)
Collecting click>=5.1 (from flask>=0.12.2->parameter-sherpa)
Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
|████████████████████████████████| 81kB 790kB/s
Collecting itsdangerous>=0.24 (from flask>=0.12.2->parameter-sherpa)
Downloading https://[PROXY-TO-PYPI]/pypi/pypi-releases/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Requirement already satisfied: Jinja2>=2.10.1 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from flask>=0.12.2->parameter-sherpa) (2.10.1)
Requirement already satisfied: Werkzeug>=0.15 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from flask>=0.12.2->parameter-sherpa) (0.16.0)
Requirement already satisfied: six>=1.5 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from python-dateutil>=2.6.1->pandas>=0.20.3->parameter-sherpa) (1.12.0)
Requirement already satisfied: MarkupSafe>=0.23 in /localdata/local/anaconda3/envs/my-conda-env/lib/python3.6/site-packages (from Jinja2>=2.10.1->flask>=0.12.2->parameter-sherpa) (1.1.1)
Building wheels for collected packages: parameter-sherpa, drmaa
Building wheel for parameter-sherpa (setup.py) ... done
Created wheel for parameter-sherpa: filename=parameter_sherpa-1.0.5-py2.py3-none-any.whl size=542235 sha256=05441754f58915f1c4f3ed058c4e5604a42995d4867a35163e2e9cfcb71042a8
Stored in directory: /net/ent-home/home/s/sv182c/.cache/pip/wheels/6e/37/de/811d40a6b17c2241f25b5cbfe55cf966ff3fe0b6c5154e3445
Building wheel for drmaa (setup.py) ... done
Created wheel for drmaa: filename=drmaa-0.7.9-py2.py3-none-any.whl size=19751 sha256=3db54bb8c6b45aa2060eaf1527350fb50258b21e98c17a7db5b1008105c1e1e2
Stored in directory: /net/ent-home/home/s/sv182c/.cache/pip/wheels/2e/fb/8c/e4209fe365b3e0ba15f30ebe1aadb7a2a8ad9720a678d1539e
Successfully built parameter-sherpa drmaa
Installing collected packages: click, itsdangerous, flask, drmaa, parameter-sherpa
Successfully installed click-7.0 drmaa-0.7.9 flask-1.1.1 itsdangerous-1.1.0 parameter-sherpa-1.0.5
So it looks like these dependencies were not able to be resolved when installing via pip:
But these dependencies could be resolved when installing via pip:
Lastly, these dependencies are "unknown" as to whether they will be resolved, since my environment happened to already have them:
I realize that the pip installation docs say:
since the source is regularly updated we recommend to clone from GitHub as described below
However, I would like to quickly and easily deploy this to an easily replicatable environment via the pip package manager, and installing via GitHub is not practical.
Is there any way that you could please look into this? (and, of course, the fix would need to be published back to the PyPI)
Why is it not possible to specify max_num_trials for the PBT algorithm or the Successive Halving algorithm? When will these algorithms be completed for an experiment?
https://parameter-sherpa.readthedocs.io/en/latest/algorithms/algorithms.html
Thanks.
The requirements.txt file and setup.py will always install the enum34 package.
From what I understand about the enum34 package, it backports the Python 3.4 Enum to older versions of Python (3.3, 3.2, 3.1, 2.7, 2.6, 2.5, and 2.4).
Therefore, if the user is running with a new enough version of Python, the enum34 package is not required.
Most importantly, having enum34 installed into a Python 3.4+ environment will cause problems since they are installed to the same location.
In the requirements.txt file, enum34 could easily be skipped for newer Python versions:
enum34; python_version < '3.4'
In the setup.py file, the Python version could easily be read from the sys.version_info tuple.
(As a separate question, why are the install requirements listed in both requirements.txt and setup.py?)
Currently logs are output, but when something crashes Sherpa points to output_dir/jobs/trial_<trial_id>.log as reference to look for bugs. Either LocalScheduler needs to write its output to those files or the message needs to be changed. Whether it also writes to the command-line output can be decided.
I'm running Sherpa on a Spark cluster (I've written a little Spark scheduler), and the default logging level is causing my Spark jobs to hang.
I've fixed it for my case by changing the logging options in each of the source files to
logging.basicConfig(level=logging.WARNING)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)I have been trying to use Sherpa bayesian opt in low level tensorflow API but no success so far.
It would be helpful, if you could update your README with an example of using tensorflow API's.
Peace to everyone
I am a beginner working on keras at colab
I know that my model is working because I see the training when setting verbose to 1
http://172.28.0.2:8880 : This site can’t be reached 172.28.0.2 took too long to respond
http://localhost:8880 : This site can’t be reached localhost refused to connect.
I tried reading :
https://cloud.google.com/deep-learning-vm/docs/jupyter
https://research.google.com/colaboratory/local-runtimes.html
but I don't know what the matter is exactly about and what to do
another question please
after creating sherpa_study.csv at my google drive and when trying
study.save(output_dir='drive/My Drive/pyton/sherpa_study.csv')
I get
NotADirectoryError: [Errno 20] Not a directory: 'drive/My Drive/pyton/sherpa_study.csv/config.pkl'
finally I want to say that your library is really awesome. I tried many other tools with my functional api multi-output regression model without success. your library seems to work and the basic part is easy to use.
Hello, I am trying to optimize the hyperparameters of 3 separate models, with each one being done on a completely separate machine. While they are completely separate models, they are made from the same codebase and are using the same hyperparameters and the same search space.
Each one is trying a RandomSearch, but there appears to be neither a search nor any randomness.
Here are screenshots from the dashboards of each machine. You can see that the parameters delivered to each trial are the same between trials and between machines! (other than the first trial)
Machine 1:
Machine 2:
Machine 3:
All 3 machines happen to be "linked" to one another by virtue of them all mounting multiple, common NFS shares. One of those happens to be where the $HOME directory is located. I have been sleuthing around to see how these processes might be intercommunicating, such as caching data somewhere under $HOME, but I have yet to find anything.
Note that I am not trying to do anything related to the "parallel" feature. Just want to run 3 separate studies at the same time.
Any insight into how I can fix this would be greatly appreciated.
Environment (on all 3 machines):
GPL is viral, which is inacceptable.
Hi -- Is there a way to show the dashboard after the run is complete? Currently, once my optimization finishes, the dashboard turns off and I'm not able to see it anymore -- presumably there's some command to re-start the dashboard?
Thanks!
I was able to get this to work in Kaggle Kernel. However, I hit a road block trying to figure out how to resolve ValueError : Trial 10 does not exist or did not submit metrics.
Any assistance would definitely help. I used this notebook as a guide.
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
3077 try:
-> 3078 return self._engine.get_loc(key)
3079 except KeyError:
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
KeyError: 'Trial-ID'
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
/opt/conda/lib/python3.6/site-packages/sherpa/core.py in finalize(self, trial, status)
148 try:
--> 149 rows = self.results.loc[self.results['Trial-ID'] == trial.id]
150 if len(rows) == 0:
/opt/conda/lib/python3.6/site-packages/pandas/core/frame.py in __getitem__(self, key)
2687 else:
-> 2688 return self._getitem_column(key)
2689
/opt/conda/lib/python3.6/site-packages/pandas/core/frame.py in _getitem_column(self, key)
2694 if self.columns.is_unique:
-> 2695 return self._get_item_cache(key)
2696
/opt/conda/lib/python3.6/site-packages/pandas/core/generic.py in _get_item_cache(self, item)
2488 if res is None:
-> 2489 values = self._data.get(item)
2490 res = self._box_item_values(item, values)
/opt/conda/lib/python3.6/site-packages/pandas/core/internals.py in get(self, item, fastpath)
4114 if not isna(item):
-> 4115 loc = self.items.get_loc(item)
4116 else:
/opt/conda/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
3079 except KeyError:
-> 3080 return self._engine.get_loc(self._maybe_cast_indexer(key))
3081
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
KeyError: 'Trial-ID'
During handling of the above exception, another exception occurred:
ValueError Traceback (most recent call last)
<ipython-input-27-8694891612ba> in <module>()
----> 1 study.finalize(trial=trial)
2
3 print(study.get_best_result())
/opt/conda/lib/python3.6/site-packages/sherpa/core.py in finalize(self, trial, status)
152 except KeyError:
153 raise ValueError("Trial {} does not exist or did not "
--> 154 "submit metrics.".format(trial.id))
155
156 # Find best row as minimum or maximum objective
ValueError: Trial 10 does not exist or did not submit metrics.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.