boschresearch/blackboxopt

Blackbox optimization algorithms with a common interface, along with useful helpers like parallel optimization loops, analysis and visualization scripts.

Blackbox Optimization

Various blackbox optimization algorithms with a common interface along with useful helpers like parallel optimization loops, analysis and visualization scripts.

Random search is provided as an example optimizer along with tests for the interface.

New optimizers can require blackboxopt as a dependency, which is just the light-weight interface definition. If you want all optimizer implementations that come with this package, install blackboxopt[all] Alternatively, you can get individual optimizers with e.g. blackboxopt[bohb]

This software is a research prototype. The software is not ready for production use. It has neither been developed nor tested for a specific use case. However, the license conditions of the applicable Open Source licenses allow you to adapt the software to your needs. Before using it in a safety relevant setting, make sure that the software fulfills your requirements and adjust it according to any applicable safety standards (e.g. ISO 26262).

Documentation

Visit boschresearch.github.io/blackboxopt

Development

Install poetry >= 1.5.0

pip install --upgrade poetry

Install the blackboxopt package from source by running the following from the root directory of this repository

poetry install

(Optional) Install pre-commit hooks to check code standards before committing changes:

poetry run pre-commit install

Test

Make sure to install all extras before running tests

poetry install -E testing
poetry run pytest tests/

For HTML test coverage reports run

poetry run pytest tests/ --cov --cov-report html:htmlcov

Building Documentation

Make sure to install all necessary dependencies:

poetry install --extras=all

The documentation can be built from the repository root as follows:

poetry run mkdocs build --clean --no-directory-urls

For serving it locally while working on the documentation run:

poetry run mkdocs serve

Architectural Decision Records

Create evaluation result from specification

In the context of initializing an evaluation result from a specification, facing the concern that having a constructor with a specification argument while the specification attributes end up as toplevel attributes and not summarized under a specification attribute we decided for unpacking the evaluation specification like a dictionary into the result constructor to prevent the said cognitive dissonance, accepting that the unpacking operator can feel unintuitive and that users might tend to matching the attributes explictly to the init arguments.

Report multiple evaluations

In the context of many optimizers just sequentally reporting the individual evaluations when multiple evaluations are reported at once and thus not leveraging any batch reporting benefits, facing the concern that representing that common behaviour in the optimizer base class requires the definition of an abstract report single and an abstract report multi method for which the report single does not need to be implemented if the report multi is, we decided to refactor the arising redundancy into a function call_functions_with_evaluations_and_collect_errors, accepting that this increases the cognitive load when reading the code.

License

blackboxopt is open-sourced under the Apache-2.0 license. See the LICENSE file for details.

For a list of other open source components included in blackboxopt, see the file 3rd-party-licenses.txt.

blackboxopt's Issues

Passing Actions workflow despite failing tests

With our current

 poetry run pytest --cov=blackboxopt tests | sed -n '/ -----------/,$p' > coverage.txt

failing tests don't cause the GH Actions workflow step to fail anymore.

Enable coverage comparison

With the coverage reporting action we still get the following note:

No coverage data of the default branch was found for comparison. A possible reason for this is that the coverage action has not yet run after a push event and the data is therefore not yet initialized.

Enable storing the coverage in the respective branch to allow the comparison in each PR.

Could it be that the following line points to a non existing step output because the step names changed between the example and our implementation?
https://github.com/boschresearch/blackboxopt/blob/main/.github/workflows/python.yaml#L85

Make Visualizer objective aware

Add support for objective (min/max directionality, objective names) in Visualizer.

Revisit ADR on Torch CPU/GPU after Poetry 1.5

With the possibility to define exclusive indexes for poetry starting from version 1.5.0 (python-poetry/poetry/issues/6713 & python-poetry/poetry/issues/7853), let's revisit our ADR on the matter and potentially switch to cpu torch by default.

objectives_matrix visualization breaks for non-multi-fidelity plots

The function to generate the hover text currently assumes to have fidelities to display, which must not be the case. Needs to be implemented more robustly.

Failing example evaluations:

evaluation = [
                EvaluationSpecification(
                    evaluation_spec_id="1", run_id="1", configuration={"p1": 1.23}
                ),
                Evaluation(
                    evaluation_spec_id="2",
                    run_id="1",
                    configuration={"p1": 1.4},
                    objectives={"loss": 1.1, "score": 9001},
                ),
                Evaluation(
                    evaluation_spec_id="3",
                    run_id="1",
                    configuration={"p1": 1.3},
                    objectives={"loss": 1.4, "score": 9001},
                ),
            ]

BOHB is using ParameterSpace features

Our linter detected an issue with BOHB's usage of the ParameterSpace:

As the type hints indicate, BOHB requires a SearchSpace for the initialization (L125).

On the other hand, it uses search_space.__iter__() when it iterates over the parameters (L180).

But the base class SearchSpace currently doesn't require an __iter__() implementation. Only the child class does.

We should check how we can solve this. (For the time being, the linter has been set to ignore the issue. But we should enable it again after we solved this)

Get __version__ from importlib.metadata

Do the following in __init__.py instead of duplicating the version information:

__version__ = importlib.metadata.version("blackboxopt")

JSON dump in optimization loop logging fails for some user_info values