This repository contains the code of Hyperparameter Optimization via Interacting with Probabilistic Circuits
.
Despite the growing interest in designing truly interactive hyperparameter optimization (HPO) methods, only a few allow human feedback to be included. However, these methods add friction to the interactive process, rigidly requiring users to define prior distributions ex ante and often imposing additional constraints on the optimization framework. This hinders flexible incorporation of expertise and valuable knowledge of domain experts, who might provide partial feedback at any time during optimization. To overcome these limitations, we introduce a novel Bayesian optimization approach leveraging probabilistic circuits (PCs) as a surrogate model. PCs encode a tractable joint distribution over the hybrid hyperparameter space and evaluation scores. They enable tractable and exact conditional inference and sampling, allowing users to provide beliefs interactively and generate configurations adhering to their feedback. We demonstrate the benefits of the resulting interactive HPO through an extensive empirical evaluation of diverse benchmarks, including the challenging setting of neural architecture search.
We recommend to run the code in a Docker
container, we don't guarantee a working version without Docker.
To setup the code and run experiments do the following instructions:
- Navigate to the root directory of the repository
- Run
docker build --build-arg baseimage=[BASE_IMAGE] --no-cache --tag ihpo .
whereBASE_IMAGE=python:3.9-slim
depending on whether you'd like to use GPU or not. - Run
docker run -v .:/app/ -it ihpo
to enter the Docker container. - Run
python main.py --exp nas101 --optimizer pc
to run the code (change options accordingly).
To simulate user interactions we allow you to put json-files into the interventions
directory. The json-files should have the following structure:
[
{
"type": "good" or "bad",
"kind": "point" or "dist"
"intervention": {YOUR INTERVENTION},
"iteration" ITERATION TO TRIGGER INTERACTION
},
...
]
You can put as many interactions as you like into the json-file. The interactions will be evaluated top down, i.e. the first appearing in the list first and so on.
The iteration
field only holds information for the optimizer when to use the interaction but it has no effect on the order of how the interactions are triggered. This means that interactions have to be ordered correctly in the list! Otherwise this might cause errors or unexpected behavior.
The "type" is only meta-data and specifies whether an interaction is intended to be beneficial or harmful. The "kind" field defines whether the interaction is a point-interaction, i.e. the user fixes a certain hyperparameter to a certain value or whether the interaction is a distribution, i.e. the user specifies a distribution over possible interactions. The "intervention" field then specifies the exact interaction as a dictionary where {"hyperparameter": VALUE or {"dist:" ["uniform, cat, gauss"], "parameters": [parameters]}}" is the syntax. The "iteration" field specifies the iteration the interaction is applied. See the existing json-files in the interventions
-directory for examples.
To plot the results of the optimization runs, swith to the directory ihpo/utils
. Here you can execute python plot.py
with several arguments (described in the argument definitions). Generating some plots can take a while (the regret_vs_cost up to 5-10 minutes).
Sometimes you might encounter an error of scikit-learn stating that some metric(s) cannot be imported. There is a hacky workaround that resolves this issue:
pip uninstall scikit-learn pandas
pip install scikit-learn pandas
pip uninstall scikit-learn
pip install scikit-learn==1.1.3
This forces pip to downgrade to scikit-learn==1.1.3.
while keeping the other packages at higher versions (might cause warnings by pip). The code should work after that.
If the automatic setup of the dataset fails, try to manually run the following command in your docker container:
gdown 1KSjP3-5x1O0_2-DuQJgcXG3_h_AiYh0j --folder -O /app/data/
If it does not succeed, try to download the contents of the Google Drive directory with the given ID (1KSjP3-5x1O0_2-DuQJgcXG3_h_AiYh0j
) manually one by one and place it in the /app/data/
directory.