pablormier/yabox

Yet another black-box optimization library for Python

Yet another black-box optimization library for Python

Description

Yabox is a very small library for black-box (derivative free) optimization of functions that only depends on numpy and matplotlib for visualization. The library includes different stochastic algorithms for minimizing a function f(X) that does not need to have an analytical form, where X = {x1, ..., xN}. The current version of the library includes the Differential Evolution algorithm and a modified version for parallel evaluation.

Example of minimization of the Ackley function (using Yabox and Differential Evolution):

Installation

Yabox is in PyPI so you can use the following command to install the latest released version:

pip install yabox

Basic usage

Pre-defined functions

Yabox includes some default benchmark functions used in black-box optimization, available in the package yabox.problems. These functions also include 2D and 3D plotting capabilities:

>>> from yabox.problems import Levy
>>> problem = Levy()
>>> problem.plot3d()

A problem is just a function that can be evaluated for a given X:

>>> problem(np.array([1,1,1]))
0.80668910823394901

Optimization

Simple example minimizing a function of one variable x using Differential Evolution, searching between -10 <= x <= 10:

>>> from yabox import DE
>>> DE(lambda x: sum(x**2), [(-10, 10)]).solve()
(array([ 0.]), 0.0)

Example using Differential Evolution and showing progress (requires tqdm)

Yabox includes a parallel version of Differential Evolution. Import PDE instead of DE:

>>> from yabox import PDE
>>> PDE(lambda x: sum(x**2), [(-10, 10)]).solve()
(array([ 0.]), 0.0)

For more examples, check the notebooks included in the project

About

This library is inspired in the scipy's differential evolution implementation. The main goal of Yabox is to include a larger set of stochastic black-box optimization algorithms plus many utilities, all in a small library with minimal dependencies.

yabox's Issues

How to pass arguments to function?

Hi,

How to pass arguments to the function? So variables that are not being optimized but they are part of the function and they help to define the problem.

Implement stop condition by convergence

Make axis3d panes transparent in plot3d

Implement rotated hyper-ellipsoid function

https://www.sfu.ca/~ssurjano/rothyp.html

Is there anyway to state a tolerance to accepted the minimum

the code is running for 20 hours already and I would accept the answer but I can not.

Modify the DE iterator to yield status on each individual operation

This would make easier to access the state of the search to collect information about the values of CR and F used, the last mutated individual, and so on.

Improve performance using numpy

Repairs, crossovers and samples can be improved using numpy functions

Typo in de.py

Your codes are very helpful to solve my projects.

While I applied your code, I found a typo that you might know already.

Typo part:
de.py -> class DE -> def solve

'return' might be changed as follows:

Original code:
return self.denormalize(P[idx].reshape(-1, 1)), fitness[idx]

Fixed:
return self.denormalize(P[idx].reshape(1, -1)), fitness[idx]

Algorithm iterator auto-stop on maxiters

Remove instance of numpy RandomState in DE/PDE classes

Use np.random instead, and np.random.seed(value) when a seed is provided in DE / PDE constructors. This will make it easier to use Numba in the future

Implement Schwefel function

https://www.sfu.ca/~ssurjano/schwef.html

Cannot pickle exception

  File "******\Anaconda3\lib\site-packages\yabox\algorithms\de.py", line 183, in solve
    for step in iterator:
  File "******\Anaconda3\lib\site-packages\yabox\algorithms\de.py", line 211, in iterator
    it = PDEIterator(self)
  File "******\Anaconda3\lib\site-packages\yabox\algorithms\de.py", line 70, in __init__
    super().__init__(de)
  File "******\Anaconda3\lib\site-packages\yabox\algorithms\de.py", line 9, in __init__
    self.fitness = de.evaluate(self.population)
  File "******\Anaconda3\lib\site-packages\yabox\algorithms\de.py", line 161, in evaluate
    return self.evaluate_denormalized(PD)
  File "******\Anaconda3\lib\site-packages\yabox\algorithms\de.py", line 221, in evaluate_denormalized
    return list(self.pool.map(self.fobj, PD, chunksize=self.chunksize))
  File "******\Anaconda3\lib\multiprocessing\pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "******\Anaconda3\lib\multiprocessing\pool.py", line 644, in get
    raise self._value
  File "******\Anaconda3\lib\multiprocessing\pool.py", line 424, in _handle_tasks
    put(task)
  File "******\Anaconda3\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "******\Anaconda3\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'Yabox.invokeScoring.<locals>.yaboxScore'

==========================================================

Implement Styblinski-Tang function

https://www.sfu.ca/~ssurjano/stybtang.html

ModuleNotFoundError: No module named 'yabox.utils'

Hi Pablormier

I am using version 1.1.0. I cannot find 1.1.1. With 1.1.0 I got an error with ModuleNotFoundError: No module named 'yabox.utils'.

Please help me on this issue.

Thans

AttributeError: Can't pickle local object 'create_lossOdeint.<locals>.lossOdeint'

I got this error when trying to use PDE

Traceback (most recent call last):
  File "/home/ats4i/Desktop/corona/dataAndModelsCovid19/dataFit_SEAIRD_v2YaboxEvolutionaryParams.py", line 1036, in <module>
    main(countriesExt)
  File "/home/ats4i/Desktop/corona/dataAndModelsCovid19/dataFit_SEAIRD_v2YaboxEvolutionaryParams.py", line 660, in main
    results = ray.get(results)
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 2349, in get
    raise value
ray.exceptions.RayTaskError: ray_Learner:train() (pid=117047, host=jedha)
  File "/home/ats4i/Desktop/corona/dataAndModelsCovid19/dataFit_SEAIRD_v2YaboxEvolutionaryParams.py", line 337, in train
    for step in de.geniterator():
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/yabox/algorithms/de.py", line 172, in geniterator
    for step in it:
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/yabox/algorithms/de.py", line 203, in iterator
    it = PDEIterator(self)
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/yabox/algorithms/de.py", line 70, in __init__
    super().__init__(de)
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/yabox/algorithms/de.py", line 9, in __init__
    self.fitness = de.evaluate(self.population)
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/yabox/algorithms/de.py", line 161, in evaluate
    return self.evaluate_denormalized(PD)
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/yabox/algorithms/de.py", line 213, in evaluate_denormalized
    return list(self.pool.map(self.fobj, PD, chunksize=self.chunksize))
  File "/home/ats4i/anaconda3/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/ats4i/anaconda3/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/home/ats4i/anaconda3/lib/python3.7/multiprocessing/pool.py", line 431, in _handle_tasks
    put(task)
  File "/home/ats4i/anaconda3/lib/python3.7/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/home/ats4i/anaconda3/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)

the function to be minimized

def create_lossOdeint(data, recovered, \
            death, s_0, e_0, a_0, i_0, r_0, d_0, startNCases, \
                 weigthCases, weigthRecov):
    def lossOdeint(point):
        size = len(data)
        beta, beta2, sigma, sigma2, sigma3, gamma, b, mu = point
        def SEAIRD(y,t):
            S = y[0]
            E = y[1]
            A = y[2]
            I = y[3]
            R = y[4]
            p=0.2
            # beta2=beta
            y0=-(beta2*A+beta*I)*S+mu*S #S
            y1=(beta2*A+beta*I)*S-sigma*E-mu*E #E
            y2=sigma*E*(1-p)-gamma*A-mu*A #A
            y3=sigma*E*p-gamma*I-sigma2*I-sigma3*I-mu*I #I
            y4=b*I+gamma*A+sigma2*I-mu*R #R
            y5=(-(y0+y1+y2+y3+y4)) #D
            return [y0,y1,y2,y3,y4,y5]

        y0=[s_0,e_0,a_0,i_0,r_0,d_0]
        tspan=np.arange(0, size, 1)
        res=odeint(SEAIRD,y0,tspan) 
        #,hmax=0.01)

        tot=0
        l1=0
        l2=0
        l3=0
        for i in range(0,len(data.values)):
            if data.values[i]>startNCases:
                l1 = l1+(res[i,3] - data.values[i])**2
                l2 = l2+(res[i,5] - death.values[i])**2
                l3 = l3+(res[i,4] - recovered.values[i])**2
                tot+=1
        l1=np.sqrt(l1/max(1,tot))
        l2=np.sqrt(l2/max(1,tot))
        l3=np.sqrt(l3/max(1,tot))
        
        #weight for cases
        u = weigthCases
        #weight for recovered
        w = weigthRecov 
        #weight for deaths
        v = max(0,1. - u - w)
        return u*l1 + v*l2 + w*l3 
    return lossOdeint

the call to minimizer

        bounds=[(1e-12, .2),(1e-12, .2),(1/60 ,0.4),(1/60, .4),
        (1/60, .4),(1e-12, .4),(1e-12, .4),(1e-12, .4)]

        maxiterations=1000
        f=create_lossOdeint(self.data, self.recovered, \
            self.death, self.s_0, self.e_0, self.a_0, self.i_0, self.r_0, self.d_0, self.startNCases, \
                 self.weigthCases, self.weigthRecov)
        de = PDE(f, bounds, maxiters=maxiterations)
        i=0
        with tqdm(total=maxiterations*500) as pbar:
            for step in de.geniterator():
                idx = step.best_idx
                norm_vector = step.population[idx]
                best_params = de.denormalize([norm_vector])
                pbar.update(i)
                i+=1
        p=best_params[0]

Implement Griewank function

https://www.sfu.ca/~ssurjano/griewank.html

Implement Langermann function

https://www.sfu.ca/~ssurjano/langer.html

Create notebook to compare the performance of Yabox vs Scipy DE

Compare vanilla DE vs DE w/parallel evaluation (and) self adaptive parameters

problem import error from yabox

Dear pablomier, I have installed yabox using the command "pip install yabox". I tried the following code to import problem from yabox. But i could not import it is showing the following error,

from yabox import DE

de=DE(problem,problem.bounds,mutation=(0.5,1.0),maxiters=1000)
Traceback (most recent call last):
File "<pyshell#5>", line 1, in
de=DE(problem,problem.bounds,mutation=(0.5,1.0),maxiters=1000)
NameError: name 'problem' is not defined

from . import ft2font

ImportError: DLL load failed while importing ft2font: The specified module could not be found.

Please help me to clear this issue.
Thank you

Using scipy's DE genetic algorithm for initial parameter estimation in non-linear regression

I see on github that you are writing optimization code using Differential Evolution for yabox. When using non-linear solvers such as Levenberg-Marquardt or gradient descent to optimize parameters for non-linear regression, there is a general problem of initial starting parameters. With poor initial parameter values to start the "descent" into error space, the algorithms can stop in a local error minimum. For this reason, the authors of scipy have added a (serial) implementation of DE for initial parameter estimation to use with scipy's non-linear solvers. The module is named scipy.optimize.differential_evolution. Scipy's implementation uses the Latin Hypercube algorithm to ensure a thorough search of parameter space.

I have used scipy's Differential Evolution genetic algorithm to determine initial parameters for fitting a double Lorentzian peak equation to Raman spectroscopy data of carbon nanotubes and found that the results were excellent. The GitHub project, with a test spectroscopy data file, is:

https://github.com/zunzun/RamanSpectroscopyFit

My background is in nuclear engineering and industrial radiation physics, and I love Python, so if you have any questions please let me know.

James Phillips

an error with PDE - pickling error

Hi Pablormier

I ran your example with parallel process:
from yabox import PDE
PDE(lambda x: sum(x2), [(-10, 10)]).solve()**

Then I got an error as below:
PicklingError: Can't pickle <function at 0x7f4ef807e268>: attribute lookup on main failed

The code continue running after maxiterations is reached

Is there anyway to stop the calculations at max iterations?

Include Latin Hypercube Sampling initialization method

Implement Levy function

https://www.sfu.ca/~ssurjano/levy.html

Auto-terminate pool after iterating on PDE

Yabox Import error

C:\Users\muham>pip install yabox
Requirement already satisfied: yabox in c:\users\muham\appdata\local\programs\python\python38-32\lib\site-packages (1.1.0)
Requirement already satisfied: numpy>=1.7 in c:\users\muham\appdata\local\programs\python\python38-32\lib\site-packages (from yabox) (1.18.5)

C:\Users\muham>python
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

from yabox.problems import problem
Traceback (most recent call last):
File "", line 1, in
ImportError: cannot import name 'problem' from 'yabox.problems' (C:\Users\muham\AppData\Local\Programs\Python\Python38-32\lib\site-packages\yabox\problems_init_.py)

Hi Sir

I hope you will be doing fine.

I have copied this from my terminal. As you can see that I have successfully installed Yabox but when I run the command from yabox.problems import problem it gives me an error.

If you can guide me I will be very grateful to you.

pass variable parameters

The code below only applies when you have constant parameters.
They are defined inside the function. No problem.

def fun(x, a, b):
    return a + sum(x**2) / (b + 1)

def f(x):
    # You can initialize here your params a, b
    return fun(x, 10, -2)

DE(f, [(-10, 10)], maxiters=1000).solve(show_progress=True)

But I have variable parameters. It varies according to the country to define the ODE for COVID-19 infection rate, death rate and recovered rates.

https://github.com/gasilva/dataAndModelsCovid19

So I need to pass the parameters at the begining of the run. They are the initial conditions for ODE solver and some other control parameters.

Implement Michalewicz function

https://www.sfu.ca/~ssurjano/michal.html

Merge _eval and _eval_population into one function

Add self-adaptive capabilities

Regression issue (fc5c5ca)

Parallel-evaluated DE does not work with lambda expressions

Since lambdas are not pickable, the Pool.map function throws an exception in Parallel DE alg.

Yabox doesn't return

https://github.com/pablormier/yabox/blob/master/yabox/algorithms/de.py#L192

that line should have 2 indentations less and break in the loop

Do not initialize Pool in PDE when processes = 0

Self-adaptive parameters

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I got this error when executing an minimization operation with yabox.
Usually this is caused by boolean or logical operation. Pandas is a suspect.

Traceback (most recent call last):
  File "/home/ats4i/Desktop/corona/dataAndModelsCovid19/adjustInitalConditionsS0DateI0WcasesWrecYabox.py", line 157, in <module>
    optimal.append(opt(country,e0,a0,r0,d0,date,version))  
  File "/home/ats4i/Desktop/corona/dataAndModelsCovid19/adjustInitalConditionsS0DateI0WcasesWrecYabox.py", line 57, in opt
    for step in de.geniterator():
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/yabox/algorithms/de.py", line 170, in geniterator
    it = self.iterator()
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/yabox/algorithms/de.py", line 167, in iterator
    return iter(DEIterator(self))
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/yabox/algorithms/de.py", line 10, in __init__
    self.best_fitness = np.minimum(self.fitness)
  File "/home/ats4i/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 1555, in __nonzero__
    self.__class__.__name__
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Implement Dixon-Price function

https://www.sfu.ca/~ssurjano/dixonpr.html

Implement Perm 0, D, Beta function

https://www.sfu.ca/~ssurjano/perm0db.html

Use mutation interval provided in DE and PDE when self_adaptive = True

What is returned by solve?

https://github.com/pablormier/yabox/blob/master/yabox/algorithms/de.py#L192

Why not to return P[idx]?

Implement Zakharov function

https://www.sfu.ca/~ssurjano/zakharov.html

Implement Perm function D, Beta

https://www.sfu.ca/~ssurjano/permdb.html

Add option in plot2D of class Problem to use plot.contour instead of imshow