vthorey/benderopt

Black-box optimization library

benderopt

benderopt is a black box optimization library.

For asynchronous use, a web client using this library is available in open access at bender.dreem.com

The algorithm implemented "parzen_estimator" is similar to TPE described in: Bergstra, James S., et al. “Algorithms for hyper-parameter optimization.” Advances in Neural Information Processing Systems.

Installation

pip install benderopt

or from the sources

Demo

Here is a comparison on 200 evaluations of a function we want to minimize. First a random estimator is used to select random evaluation point. Then the parzen_estimator implemented in benderopt is used to select evaluation points.

The function to minimize is the following: cos(x) + cos(2 * x + 1) + cos(y).

The red point correspond to the location of the global minima between 0 and 2pi for x and y.

The code to generate the video can be found in benchmark/benchmark_sinus2D

We can observe on this example that the parzen estimator tends to explore more the local minimum than the random approach. This might lead to a better optimization given a fixed number of evaluations.

The goal

In Black box optimization, we have a function to optimize but cannot compute the gradient, and evaluation is expensive in term of time / ressource. So we want to find a good exploration-exploitation trade off to get the best hyperparameters in as few evaluations as possible. Use case are:

• Optimization of a machine learning model (number of layers of a neural network, function of activation, etc.
• Business optimization (marketing campain, a/b testing)

Code Minimal Example

One of the advantage of benderopt is that it uses JSON-like object representation making it easier for a user to define parameters to optimize. This also allows an easy to integratation with an asynchrounous system such as bender.dreem.com.

Here is a minimal example.

from benderopt import minimize
import numpy as np
import logging

logging.basicConfig(level=logging.DEBUG) # logging.INFO will print less information

# We want to minimize the sinus function between 0 and 2pi
def f(x):
    return np.sin(x)

# We define the parameters we want to optimize:
optimization_problem_parameters = [
    {
        "name": "x", 
        "category": "uniform",
        "search_space": {
            "low": 0,
            "high": 2 * np.pi,
        }
    }
]

# We launch the optimization
best_sample = minimize(f, optimization_problem_parameters, number_of_evaluation=50)

print(best_sample["x"], 3 * np.pi / 2)


> 4.710390692396651 4.71238898038469

Minimal Documentation:

Optimization Problem

An optimization problem contains:

• A list of parameters (i.e. parameters with their search space)
• A list of observation (i.e. values for each parameter of the list and a corresponding loss)

We use JSON-like representation for each of them e.g.

optimization_problem_data = {
    "parameters": [
        {
             "name": "parameter_1", 
             "category": "uniform",
             "search_space": {"low": 0, "high": 2 * np.pi, "step": 0.1}
        },
        {
            "name": "parameter_2", 
            "category": "categorical",
            "search_space": {"values": ["a", "b", "c"]}
        }
    ],
    "observations": [
        {
            "sample": {"parameter_1": 0.4, "parameter_2": "a"},
            "loss": 0.08
        },
        {
            "sample": {"parameter_1": 3.4, "parameter_2": "a"},
            "loss": 0.1
        },
        {
            "sample": {"parameter_1": 4.1, "parameter_2": "c"},
            "loss": 0.45
        },
    ]
}

Optimizer

An optimizer takes an optimization problem and suggest new_predictions. In other words, an optimizer takes a list of parameters with their search space and a history of past evaluations to suggest a new one.

Using the optimization_problem_data from the previous example:

from benderopt.base import OptimizationProblem, Observation
from benderopt.optimizer import optimizers

optimization_problem = OptimizationProblem.from_json(optimization_problem_data)
optimizer = optimizers["parzen_estimator"](optimization_problem)
sample = optimizer.suggest()

print(sample)

> {"parameter_1": 3.9, "parameter_2": "b"}

Optimizers currently available are random and parzen_estimator.

Benderopt allows to add a new optimizer really easily by inheriting an optimizer from BaseOptimizer class.

You can check benderopt/optimizer/random.py for a minimal example.

Minimize function

Minimize function shown above in the minimal example section implementation is quite strateforward:

optimization_problem = OptimizationProblem.from_list(optimization_problem_parameters)
optimizer = optimizers["parzen_estimator"](optimization_problem)
for _ in range(number_of_evaluation):
    sample = optimizer.suggest()
    loss = f(**sample)
    observation = Observation.from_dict({"loss": loss, "sample": sample})
    optimization_problem.add_observation(observation)

The optimization_problem's observation history list is extended with a new observations at each iteration. This allows the optimizer to take them into account for the next suggestion.

Uniform Parameter

parameter	type	default	comments
low	mandatory	-	lowest possible value: all values will be greater than or equal to low
high	mandatory	-	highest value: all values will be stricly less than high
step	optionnal	None	discretize the set of possible values: all values will follow 'value = low + k * step with k belonging to [0, K]'

e.g.

    {
        "name": "x", 
        "category": "uniform",
        "search_space": {
            "low": 0,
            "high": 2 * np.pi,
            # "step": np.pi / 8
        }
    }

Log-Uniform Parameter

parameter	type	default	comments
low	mandatory	-	lowest possible value: all values will be greater than or equal to low
high	mandatory	-	highest value: all values will be stricly less than high
step	optionnal	None	"discretize the set of possible values: all values will follow 'value = low + k * step with k belonging to [0, K]'"
base	optional	10	logarithmic base to use

e.g.

    {
        "name": "x", 
        "category": "loguniform",
        "search_space": {
            "low": 1e-4,
            "high": 1e-2,
            # "step": 1e-5,
            # "base": 10,
        }
    }

Normal Parameter

parameter	type	default	comments
low	optionnal	-inf	lowest possible value: all values will be greater than or equal to low
high	optionnal	inf	highest value: all values will be stricly less than high
mu	mandatory	-	mean value: all values will be initially drawn following a gaussian centered at mu with sigma variance
sigma	mandatory	-	sigma value: all values will be initially drawn following a gaussian centered at mu with sigma variance
step	optionnal	None	"discretize the set of possible values: all values will follow 'value = low + k * step with k belonging to [0, K]'"

e.g.

    {
        "name": "x", 
        "category": "normal",
        "search_space": {
            # "low": 0,
            # "high": 10,
            "mu": 5,
            "sigma": 1
            # "step": 0.01,
        }
    }

Log-Normal Parameter

parameter	type	default	comments
low	optionnal	-inf	lowest possible value: all values will be greater than or equal to low
high	optionnal	inf	highest value: all values will be stricly less than high
mu	mandatory	-	mean value: all values will be initially drawn following a gaussian centered at mu with sigma variance
sigma	mandatory	-	sigma value: all values will be initially drawn following a gaussian centered at mu with sigma variance
step	optionnal	None	"discretize the set of possible values: all values will follow 'value = low + k * step with k belonging to [0, K]'"
base	optional	10	logarithmic base to use

e.g.

    {
        "name": "x", 
        "category": "lognormal",
        "search_space": {
            # "low": 1e-6,
            # "high": 1e0,
            "mu": 1e-3,
            "sigma": 1e-2
            # "step": 1e-7,
            # "base": 10,
        }
    }

Categorical Parameter

parameter	type	default	comments
values	mandatory	-	list of categories: all values will be sampled from this list
probabilities	optionnal	number_of_values * [1 / number_of_values]	list of probabilities: all values will be initially drawn following this probability list

e.g.

    {
        "name": "x", 
        "category": "categorical",
        "search_space": {
            "values": ["a", "b", "c", "d"],
            # "probabilities": [0.1, 0.2, 0.2, 0.2, 0.3]
        }
    }

benderopt's Issues

Expose optimizers, OptimizationProblem and Observation

Hi,

Thank you so much for this package! Was looking for a hyperparameter optimizer that was both simple and easy to integrate and this just hit the nail on the head.

However, I have to access the underlying classes and methods to fit my use case via a small hack:

import benderopt.minimize as benderopt

benderopt.OptimizationProblem...

I suggest to allow more flexibility by adding these to __init__.py instead. The minimize point of access does not work for me, I imagine many people would like a slightly lower level access too.

If the decision is not to go in this direction I hope you keep the backdoor open 😛

Use custom random number generator for optimizers.

As part of my function (checking hyperparameters for NN training) I need to reset all global random number generators in python, which leads to optimizers suggesting the same value on subsequent passes.

Would be great if optimizers used their own generators.

I might create a PR in the future.

Error in np.random.choice() when sampling categorical variable consisting of size-one np.arrays()

Problem: Attempting to sample from a categorial variable, defined as a list containing np.arrays (see example below), generates an error in the np.random.choice() function used to draw a sample in function generate_samples_categorical() in ~/stats/categorical.py. The error message of the np.random.choice() function states that "a is not a 1-D variable".

control_handles_dict['Nc']['bounds_values'] = [ np.array([1.5]), np.array([2.0]), np.array([2.5]) ]

optimization_problem_parameters = [
    {
        "name": "x1",
        "category" : "categorical",
        "search_space" : {
            "values" : control_handles_dict['Nc']['bounds_values']
        }
    },
    {
        "name": "x2",
        "category" : "categorical",
        "search_space" : {
            "values" : control_handles_dict['Nc']['bounds_values']
        }
    },
]

some_obj_fun(**x):
    return x['x1'] + x['x2']

minimize(obj_fun, optimization_problem_parameters)

If one of the np.arrays() in control_handles_dict['Nc']['bounds_values'] is swapped for another data type, f.i.,

control_handles_dict['Nc']['bounds_values'] = [ 'a', np.array([2.0]), np.array([2.5]) ]

the error does not occur and np.random.choice() draws a sample from the above list.

Swapping numpy.random.choice() out for random.choices() in function generate_samples_categorical() in ~/stats/categorical.py resolves the issue.

import random

def generate_samples_categorical(values, probabilities, size=1):
    """Generate sample for categorical data with probability probabilities."""
    return random.choices(values, weights=probabilities, k=size)

Let me know if the above explanation is clear and thanks in advance.

Provide optimizer directly to minimize, rather than type.

Adding new optimizers is quite cumbersome now, since I have to modify a dictionary in your module to add them.

It would be better if minimize was also able to take optimizer directly, instead of a type.

Will probably make a PR at some point.

Minimum example doesn't work

Hi, I tried to run the file minimizer.py, however, it raised an error:

Traceback (most recent call last):
  File "/home/zippingsugar/anaconda2/envs/HLSTS/lib/python2.7/site-packages/benderopt-1.3.2-py2.7.egg/benderopt/minimizer.py", line 44, in <module>
    f, optimization_problem_parameters=optimization_problem_parameters, number_of_evaluation=100)
  File "/home/zippingsugar/anaconda2/envs/HLSTS/lib/python2.7/site-packages/benderopt-1.3.2-py2.7.egg/benderopt/minimizer.py", line 19, in minimize
    optimization_problem = OptimizationProblem.from_list(optimization_problem_parameters)
  File "/home/zippingsugar/anaconda2/envs/HLSTS/lib/python2.7/site-packages/benderopt-1.3.2-py2.7.egg/benderopt/base/optimization_problem.py", line 259, in from_list
    return cls(parameters)
  File "/home/zippingsugar/anaconda2/envs/HLSTS/lib/python2.7/site-packages/benderopt-1.3.2-py2.7.egg/benderopt/base/optimization_problem.py", line 80, in __init__
    raise ValidationError(message="'parameters' must be a list of Parameter")
benderopt.validation.utils.ValidationError: 'parameters' must be a list of Parameter

I guess there were some modifications in the code and the example wasn't updated in time.

---

Oh, I realized this error might come from using python 2.7. I've modified several places and made it compatible with py2.7.

ValidationError Message

Hi, first of all thank you for this excellent optimization package.

When setting wrong settings for an uniform parameter:

{ "name": "WestWWR", "category": "uniform", "search_space": { "low": 0.0, "high": 0.15, "step": 0.20} },

I get ValidationError: 'step' must be strictly superior than 'high'

Shouldn't it be the other way around ?
'high' must be strictly superior than 'step'

Specifying a variable is larger than another.

I'm using minimize and I have variables named x and y. Both in the closed range [0, 10]. Is it possible to give a constraint in which y is larger than x?
I can implement it on my own by returning a large value in my function but is there a more correct way to do that?

Initial guess

Hi, is there a way to provide an initial guess or starting point for the exploration? Should I just build a custom OptimizationProblem object with an observation already filled in? I tried doing this like the example using OptimizationProblem.from_json, but it seems that it won't take a dictionary, only strings or other objects.