claesenm / optunity Goto Github PK
View Code? Open in Web Editor NEWoptimization routines for hyperparameter tuning
Home Page: http://www.optunity.net
License: Other
optimization routines for hyperparameter tuning
Home Page: http://www.optunity.net
License: Other
Random search seems to ignore bounds. The problem is either in suggest_from_box
or the static key order used in function evaluation.
In Linux/mac we have a solution python -m optunity > /tmp/py2r
. However, Windows does not support that (http://superuser.com/questions/430466/in-windows-can-i-redirect-stdout-to-a-named-pipe-in-command-line).
So an option is to pass the name of the pipe to python with a paramater, like:
python -m optunity.piped -p py2r
And in python use
f = open(r'\\.\pipe\py2r', 'w', 0)
...
f.write(...)
This would be only used in windows. Seems like the easiest approach or are there other options?
Having some questions about the usage of Optunity, i was not able to find any mailing list or discussion board. Am i missing something?
Not sure what happened but builds suddenly broke on both travis & rtd. Needs fixing ASAP (tomorrow).
Seed is taken as input but not passed to Python API in generate_folds call.
A python problem: lambda is a reserved keyword. Would be nice to have a workaround. For at least API calls.
echo '{"optimize" : {"max_evals": 0}, "solver": {"solver_name" : "grid search", "lambda":[0,10]}}' | python -m optunity.piped
Exception in thread FutureThread:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "optunity/parallel.py", line 131, in Wrapper
self.__result=func(*param)
File "optunity/communication.py", line 157, in wrap
result = f(*args)
File "optunity/functions.py", line 357, in wrapped_f
return f(**dict([(k, v) for k, v in zip(keys, args)]))
File "optunity/functions.py", line 232, in wrapped_f
wrapped_f.argtuple = collections.namedtuple('args', wrapped_f.keys)
File "/usr/lib/python2.7/collections.py", line 334, in namedtuple
'keyword: %r' % name)
ValueError: Type names and field names cannot be a keyword: 'lambda'
IPython crashes if the parallelized function outputs anything to stdout or stderr. This is an IPython issue that we can't fix ourselves.
More info at: ipython/ipython#2438
A workaround is to use IPython's own parallel features:
http://nbviewer.ipython.org/github/vals/scilife-python-course/blob/master/parallel%20python.ipynb
Include DEAP in the Optunity package and automate code conversion from 2 to 3 on install (setup.py).
Note: the fact DEAP needs 2 to 3 conversion may cause issues for non-Python users (e.g. in wrappers).
An error with make_solver and random search. Expecting instead a JSON message.
echo '{"make_solver":{"solver_name":"random search"}}' | python -m optunity.piped
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/jaak/git/optunity/optunity/piped.py", line 482, in <module>
make_solver(startup_msg['make_solver'])
File "/home/jaak/git/optunity/optunity/piped.py", line 389, in make_solver
optunity.make_solver(**solver_config)
File "optunity/api.py", line 282, in make_solver
return solvercls(*args, **kwargs)
TypeError: __init__() takes exactly 2 arguments (1 given)
When using grid search the make_solver works:
echo '{"make_solver":{"solver_name":"grid search"}}' | python -m optunity.piped
{"success": "true"}
I tried to add the argument 'class_weight' to search space for maximize_structured function. Although it works well when I put a normal value such as 'class_weight':[1,10], the search space gets to be broken when I put 'None' like 'class_weight':None which is the default value for 'class_weight'.
We checked Windows setup with Dusan and found out there is an issue:
So python -m optunity.piped
will not work in Windows, but a simple solution is to call optunity.piped from a separate script:
if __name__=='__main__'
), like explained here http://www.artima.com/weblogs/viewpost.jsp?thread=4829#!/usr/bin/env python
import optunity.piped
optunity.piped.main()
Then we will just execute run.py in Windows (or also in other systems). Note: shebang is ignored in Windows.
@claesenm if the solution looks fine to you let me know, I can easily do the implementation or you can do it :).
MATLAB blocks at accepting the socket connection. Strange, because it works on other pairs of Windows/MATLAB versions.
Further testing required to diagnose the problem. Maybe due to Java.
Would be useful to have call example for optimize function in piped
http://optunity.readthedocs.org/en/latest/api/optunity.piped.html#optimize
optunity.piped works but uses way too many resources currently when doing vector evaluations (1 thread per evaluation, solvers like grid search can easily have thousands ...). Need to use concurrent.future backport in 2.x
http://optunity.readthedocs.org/en/latest/api/optunity.solvers.html#optunity.solvers.ParticleSwarm
Might be useful to have an example of ParticleSwarm. Mainly for showing that the contents of kwargs is box constraints (currently there is also no comment about kwargs under ParticleSwarm).
$jython -Dpython.path=$PYTHONPATH examples_parabola.py
Traceback (most recent call last):
File "examples_parabola.py", line 6, in <module>
optimal_pars, details, _ = optunity.maximize(f, num_evals=200, x=[-5, 5], y=[-5, 5])
File "/data/svn/claesenm/python/optunity/optunity/api.py", line 170, in maximize
solution, details = optimize(solver, f, maximize=True, max_evals=num_evals,
File "/data/svn/claesenm/python/optunity/optunity/api.py", line 244, in optimize
optimum = f.call_log[t(**solution)]
KeyError: args(x=0.04816650353569685, y=0.010200396673881285)
Maybe a rounding issue? Maybe something else.
When running 'grid search' from command line gives error when 'random search' works.
Grid search
echo '{"maximize" : {"solver_name" : "grid search", "num_evals": 5, "x":[0,10]} }' | python -m optunity.piped
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/jaak/git/optunity/optunity/piped.py", line 495, in <module>
startup_msg.get('call_log', None))
File "/home/jaak/git/optunity/optunity/piped.py", line 421, in max_or_min
solution, rslt, solver = solve_fun(func, pmap=mgr.pmap, **kwargs)
File "optunity/api.py", line 172, in maximize
pmap=pmap)
File "optunity/api.py", line 245, in optimize
optimum = f.call_log[t(**solution)]
TypeError: __new__() got an unexpected keyword argument 'x'
Random search (works)
echo '{"maximize" : {"solver_name" : "random search", "num_evals": 5, "x":[0,10]} }' | python -m optunity.piped
[{"x": 4.90398335468162}, {"x": 2.9585346492371967}, {"x": 8.75973165625038}, {"x": 0.934354531998024}, {"x": 5.092791699936537}]
{"error_msg": "Broken pipe."}
optunity.manual()
Traceback (most recent call last):
File "", line 1, in
File "/data/svn/claesenm/python/optunity/optunity/api.py", line 92, in manual
man = solver_registry._manual_lines()
AttributeError: 'module' object has no attribute '_manual_lines'
I am getting the following error by setting the aggregator option to opt.cross_validation.identity
---------------------------------------------------------------------------
33 # Define Parameter Tuning
---> 34 optimal_pars_clf_sgd, _, _ = opt.maximize(clf_sgd_cv, num_evals=n_hyperparams_evals, alpha=[0.001, .1], l1_ratio=[0., 1.])
35
36 # Train model on the Inner Training Set with Tuned Hyperparameters
../local/lib/python2.7/site-packages/optunity/api.pyc in maximize(f, num_evals, solver_name, pmap, **kwargs)
179 solver = make_solver(**suggestion)
180 solution, details = optimize(solver, f, maximize=True, max_evals=num_evals,
--> 181 pmap=pmap)
182 return solution, details, suggestion
183
../local/lib/python2.7/site-packages/optunity/api.pyc in optimize(solver, func, maximize, max_evals, pmap)
243 time = timeit.default_timer()
244 try:
--> 245 solution, report = solver.optimize(f, maximize, pmap=pmap)
246 except fun.MaximumEvaluationsException:
247 # early stopping because maximum number of evaluations is reached
../local/lib/python2.7/site-packages/optunity/solvers/ParticleSwarm.pyc in optimize(self, f, maximize, pmap)
257 fitnesses = pmap(evaluate, list(map(self.particle2dict, pop)))
258 for part, fitness in zip(pop, fitnesses):
--> 259 part.fitness = fit*fitness
260 if not part.best or part.best_fitness < part.fitness:
261 part.best = part.position
TypeError: can't multiply sequence by non-int of type 'float'
Here is my code
import optunity as opt
from optunity.metrics import _recall, contingency_table
from sklearn.linear_model import SGDClassifier
import numpy as np
n_in = 1
k_in = 2
n_hyperparams_evals = 10
clf_sgd = SGDClassifier(
penalty="elasticnet",
shuffle=True,
n_iter=500,
fit_intercept=True,
learning_rate="optimal")
# Define Inner CV
cv_decorator = opt.cross_validated(x=X, y=Y.values,
num_folds=k_in, num_iter=n_in,
strata=[Y[Y==1].index.values],
regenerate_folds=True,
aggregator=opt.cross_validation.identity)
def obj_fun_clf_sgd(x_train, y_train, x_test, y_test, alpha, l1_ratio):
model = clf_sgd.set_params(l1_ratio=l1_ratio, alpha=alpha).fit(x_train, y_train)
y_pred = model.predict(x_test)
score = _recall(contingency_table(y_test,y_pred))
return score
clf_sgd_cv = cv_decorator(obj_fun_clf_sgd)
# Define Parameter Tuning
optimal_pars_clf_sgd, _, _ = opt.maximize(clf_sgd_cv, num_evals=n_hyperparams_evals, alpha=[0.001, .1], l1_ratio=[0., 1.])
# Train model on the Inner Training Set with Tuned Hyperparameters
optimal_model_clf_sgd = clf_sgd.set_params(**optimal_pars_clf_sgd).fit(X, Y.values)
The objective is to keep track of all the scores from the various folds. Is it a bug? or am I using incorrectly the API ?
Thanks in advance
In the doc the first CV example returns 0.0. Maybe we could have a more practical example there?
http://optunity.readthedocs.org/en/latest/user/cross_validation.html
@opt.cross_validated(x=data, y=labels, num_folds=3)
def cved(x_train, y_train, x_test, y_test):
train(x_train, y_train)
predict(x_test)
return 0.0
cved()
PSO solver sometimes stops early, with less function evaluations than we allow.
In some cases PR curves are incorrect.
In optunity/api.py line 176 sys.float_info.min is set as default:
f = _wrap_hard_box_constraints(f, kwargs, sys.float_info.min)
However, sys.float_info.min is the "minimum positive normalized float" [1] which evaluates to 2.2250738585072014e-308 in my case. This causes maximize to to choose parameters that violate constraints if something like a negative mean squared error should be maximized. I suggest to replace this line by:
f = _wrap_hard_box_constraints(f, kwargs, -sys.float_info.max)
[1] https://docs.python.org/2/library/sys.html#sys.float_info
Currently nosetests only runs the tests in optunity/solvers/. For some reason those in optunity/ are being ignored.
Remove all traces of numpy dependency from optunity._cross_validation.
I think it's a matter of missing dependencies. I issued the following commands:
git clone https://github.com/claesenm/optunity.git
cd optunity/wrappers
R CMD build R
R CMD INSTALL optunity_0.6.tar.gz
Error:
* installing to library ‘/home/badn/R/x86_64-pc-linux-gnu-library/3.1’
ERROR: dependencies ‘ROCR’, ‘enrichvs’ are not available for package ‘optunity’
* removing ‘/home/badn/R/x86_64-pc-linux-gnu-library/3.1/optunity’```
sklearn has a tendency of returning numpy stuff, which in turn have a tendency not to iterate as one might expect.
I got an error when I tried to save a value of which class is 'optunity.util.NT' with pickling. My OS is windows 7 64 bit and use Python 2.7.2. I show the code below. Note that I got a similar error when I tried parallelization using optunity.pmap: pickling error.
---code
optimal_pars, info, optimal_pars2 = optunity.maximize_structured(svm_acc,search_space=args,num_evals=num_evals)
---this works (type:dictionary)
with open('test1','wb') as f:
pickle.dump(optimal_pars,f)
---this doesn't work (type:optunity.util.NT)
with open('test2','wb') as f:
pickle.dump(info,f)
---error
Traceback (most recent call last):
File "", line 2, in
File "C:\Python27\lib\pickle.py", line 1370, in dump
Pickler(file, protocol).dump(obj)
File "C:\Python27\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python27\lib\pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "C:\Python27\lib\pickle.py", line 401, in save_reduce
save(args)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 562, in save_tuple
save(element)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 748, in save_global
(obj, module, name))
PicklingError: Can't pickle <class 'optunity.util.NT'>: it's not found as optunity.util.NT
As explained here: http://stackoverflow.com/q/30539388/2148672
Typical error when I try to run any function is:-
octave:1> optunity_example
error: 'optunity' undefined near line 5 column 11
error: called from:
error: /home/andrew/my_source_makes/optunity/wrappers/matlab/optunity_example.m at line 5, column 9
Output of debug is:-
octave:2> global DEBUG_OPTUNITY
octave:3> DEBUG_OPTUNITY=true
DEBUG_OPTUNITY = 1
Al the relevant folders in the optunity directory are in Octave's path environment. Any suggestions?
If Use 'TPE' solver, the code will return below error:
optunity\solvers\TPE.pyc in algo(_args, *_kwargs)
117 def algo(_args, *_kwargs):
118 seed = self.seed if self.seed else random.randint(0, 9999999999)
--> 119 return hyperopt.tpe.suggest(_args, seed=seed *_kwargs)
120
121 space = [hyperopt.hp.uniform(k, v[0], v[1]) for k, v in self.bounds.items()]
TypeError: suggest() got multiple values for keyword argument 'seed'
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/queues.py", line 266, in _feed
send(obj)
PicklingError: Can't pickle optunity.solvers.ParticleSwarm.Particle: attribute lookup optunity.solvers.ParticleSwarm.Particle failed
Typically these loss functions are compatible with multiclass problems. To make this work, we would need to provide a more generic interface (or require preprocessing on the user end).
I'm currently leaning towards preprocessing on the user's end, e.g. have users select the predicted probability of the correct class prior to calling brier() or logloss().
We use Python's (standard) multiprocessing library for interthread communication (via multiprocessing.Queue). Unfortunately, Jython currently doesn't support this library.
A transition to something else would solve this problem.
Is it possible to tune a classifier's class weight vector using optunity?
Currently the parameter space is defined in optunity's 'minimize' and 'maximize' by passing it the lower and upper bounds. This assumes the parameter is a single number (such as 'C' or 'gamma'). I am wondering whether it is possible to optimize a weight vector, for example, where the length of the vector equals the number of classes.
Thanks
DIRECT solver not fully implemented. Should not be used yet.
Hello. When i am working on an example of Optunity+e1071 in R link, cv.setup give me an error:
Error in if (is.na(port) || port == 0) { :
missing value where TRUE/FALSE needed
How can i fix this issue?
The following list of errors should be caught at the highest level:
Hi,
installed optunity and deap through pip and tried to run git parabola.py example but got a few errors, i am using Python 3.4, Numpy 1.10.0.dev-9671491 (git-version) and DEAP 1.0.1.
with cma-es algorithm i got this error:
File "/usr/local/lib/python3.4/dist-packages/deap/cma.py", line 84, in init
self.dim = len(self.centroid)
TypeError: len() of unsized object*
After an investigation i tried to correct it by passing a list instead of a dict_values type at the CMAES.py, looked like this:
if self.lambda_:
strategy = deap.cma.Strategy(centroid=list(self.start.values()),
sigma=self.sigma, lambda_=self.lambda_)
else:
strategy = deap.cma.Strategy(centroid=list(self.start.values()),
sigma=self.sigma)
then, i got another error:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 245, in optimize
solution, report = solver.optimize(f, maximize, pmap=pmap)
File "/usr/local/lib/python3.4/dist-packages/optunity/solvers/CMAES.py", line 156, in optimize
halloffame=hof, verbose=False)
File "/usr/local/lib/python3.4/dist-packages/deap/algorithms.py", line 428, in eaGenerateUpdate
for ind, fit in zip(population, fitnesses):
File "/usr/local/lib/python3.4/dist-packages/optunity/solvers/CMAES.py", line 149, in evaluate
individual)])),)
File "/usr/local/lib/python3.4/dist-packages/optunity/functions.py", line 286, in wrapped_f
value = f(_args, *_kwargs)
File "/usr/local/lib/python3.4/dist-packages/optunity/functions.py", line 338, in wrapped_f
raise MaximumEvaluationsException(max_evals)
optunity.functions.MaximumEvaluationsException: 100During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "parabola.py", line 24, in
solver_name=solver)
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 212, in minimize
pmap=pmap)
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 254, in optimize
solution = operator.itemgetter(index)(f.call_log.keys())._asdict()
TypeError: 'KeysView' object does not support indexing
tried to run /usr/local/lib/python3.4/dist-packages/optunity/tests/test_solvers.py but got same errors:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 245, in optimize
solution, report = solver.optimize(f, maximize, pmap=pmap)
File "/usr/local/lib/python3.4/dist-packages/optunity/solvers/CMAES.py", line 156, in optimize
halloffame=hof, verbose=False)
File "/usr/local/lib/python3.4/dist-packages/deap/algorithms.py", line 428, in eaGenerateUpdate
for ind, fit in zip(population, fitnesses):
File "/usr/local/lib/python3.4/dist-packages/optunity/solvers/CMAES.py", line 149, in evaluate
individual)])),)
File "/usr/local/lib/python3.4/dist-packages/optunity/functions.py", line 286, in wrapped_f
value = f(_args, *_kwargs)
File "/usr/local/lib/python3.4/dist-packages/optunity/functions.py", line 338, in wrapped_f
raise MaximumEvaluationsException(max_evals)
optunity.functions.MaximumEvaluationsException: 100During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/optunity/tests/test_solvers.py", line 16, in
solver_name=solver)
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 181, in maximize
pmap=pmap)
File "/usr/local/lib/python3.4/dist-packages/optunity/api.py", line 254, in optimize
solution = operator.itemgetter(index)(f.call_log.keys())._asdict()
TypeError: 'KeysView' object does not support indexing
Hi,
I'd like to know if it is possible to use optunity to find hyperparameter solutions, when these parameters are discrete.
Then is it possible with non-ordinal hyperparameters ?
Finally, if it is not possible, do you plan to implement it and in case the answer is affirmative, for when ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.