fugue-project/tune

An abstraction layer for parameter tuning

Tune is an abstraction layer for general parameter tuning. It is built on Fugue so it can seamlessly run on any backend supported by Fugue, such as Spark, Dask and local.

Installation

pip install tune

It's recommended to also install Scikit-Learn (for all compatible models tuning) and Hyperopt (to enable Bayesian Optimization)

pip install tune[hyperopt,sklearn]

Quick Start

To quickly start, please go through these tutorials on Kaggle:

1. Search Space
2. Non-iterative Problems, such as Scikit-Learn model tuning
3. Iterative Problems, such as Keras model tuning

Design Philosophy

Tune does not follow Scikit-Learn's model selection APIs and does not provide distributed backend for it. We believe that parameter tuning is a general problem that is not only for machine learning, so our abstractions are built from ground up, the lower level APIs do not assume the objective is a machine learning model, while the higher level APIs are dedicated to solve specific problems, such as Scikit-Learn compatible model tuning and Keras model tuning.

Although we didn't base our solution on any of HyperOpt, Optuna, Ray Tune and Nevergrad etc., we are truly inspired by these wonderful solutions and their design. We also integrated with many of them for deeper level optimizations.

Tuning problems are never easy, here are our goals:

• Provide the simplest and most intuitive APIs for major tuning cases. We always start from real tuning cases, figure out the minimal requirement for each of them and then determine the layers of abstraction. Read this tutorial, you can see how minimal the interfaces can be.
• Be scale agnostic and platform agnostic. We want you to worry less about distributed computing, and just focus on the tuning logic itself. Built on Fugue, Tune let you develop your tuning process iteratively. You can test with small spaces on local machine, and then switch to larger spaces and run distributedly with no code change. It can effectively save time and cost and make the process fun and rewarding. And to run any tuning logic distributedly, you only need a core framework itself (Spark, Dask, etc.) and you do not need a database, a queue service or even an embeded cluster.
• Be highly extendable and flexible on lower level. For example
  • you can extend on Fugue level, for example create an execution engine for Prefect to run the tuning jobs as a Prefect workflow
  • you can integrate third party optimizers and use Tune just as a distributed orchestrator. We have integrated HyperOpt. And Optuna and Nevergrad is on the way.
  • you can start external instances (e.g. EC2 instances) for different training subtasks and to fully utilize your cloud
  • you can combine with distributed training as long as your have enough compute resource

Focuses

Here are our current focuses:

• A flexible space design and can describe a hybrid space of grid search, random search and second level optimization such as bayesian optimization
• Integrate with 3rd party tuning frameworks
• Create generalized and distributed versions of Successive Halving, Hyperband and Asynchronous Successive Halving.

Collaboration

We are looking for collaborators, if you are interested, please let us know. Please join our Slack channel.

tune's Issues

[FEATURE] Make space and parameters to be more flexible by template

Is your feature request related to a problem? Please describe.

1. Current space doesn't support non-native data types. For example what if a parameter is a pandas dataframe?
2. Current space doesn't support same expression at multiple places

For example

u = Grid(0,1)
space = Space(a=u, b=u)
list(space)

We will have 4 combinations in this case. But it is not ideal, ideally, we should have only two (dict(a=0,b=0) and dict(a=1,b=1))

In contrast, we should have 4 in the following

space = Space(a=Grid(0,1), b=Grid(0,1))
list(space)

This is because the two expressions are instantiated separately not like the first case.

In order to have the two improvements, we need to have a template idea.

A template can be a nested structure and can have the same expression at multiple places. Plus it can have non-native values, and the template doesn't need to be json serializable, it only needs to be picklable. We will use pickle to serialize in the future.

[FEATURE] Realtime charts should print the best results

Describe the solution you'd like
We need to see the current best metrics printed somewhee

[FEATURE] Increase test coverage to 100% and add docs

We were in a rush to integrate with mlflow, but the unit tests and docs were lacking.

[FEATURE] Iterative tuning for xgboost

Describe the solution you'd like
Utility function to tune xgboost models iteratively

[FEATURE] Add tests for Windows, 3.6 - 3.10

We need to expand our unit tests to windows and 3.6 - 3.10

[FEATURE] Use cloudpickle to replace json

Is your feature request related to a problem? Please describe.
Currently Trial, TrialReport TrialDecision are all json serialized. It is not necessary to use json to serialize these objects

Describe the solution you'd like
We should use cloudpickle which can handle more custom objects as well as functions (for FuncParam). It should also simplify the code.

[BUG] Reporting result is causing time out

Describe the bug
The problem is at here

tune/tune/noniterative/study.py

Line 42 in 8ec96a7

_on_report(report)

And the report contains trial, and if the trail contains dataframes, the whole dataframe is sent back causing timeout.

Expected behavior
The dataframe should be removed from the report before reporting

Environment (please complete the following information):

• Backend: pandas/dask/ray?
• Backend version:
• Python version:
• OS: linux/windows

[FEATURE] Codacy integration

Describe the solution you'd like
We need to integrate with codacy and get quality A badge
.

[FEATURE] PyTorch lightning tuning

Describe the solution you'd like
Utility function to tune pytorch lightning models iteratively

[BUG] Rand lower bound >1 when log=True

Describe the bug
When log=True, the lower bound of rand is set to 1, or error will be thrown.

Expected behavior
This is not necessary, the lower bound should be (0.0

Environment (please complete the following information):

• Backend: pandas/dask/ray?
• Backend version:
• Python version:
• OS: linux/windows

[FEATURE] TransitionChoice

Is your feature request related to a problem? Please describe.
This idea seems redundant, however, practically, this is very convenient to use.

For example, some parameter ranging from 1D to 10D

We can use RandInt instead, but where to do the conversion? In the objective? The actually breaks user logic to 2 parts. So with TransitionChoice, user define the parameter at one place.

[FEATURE] Refactor Runner to LocalOptimizer

Describe the solution you'd like
LocalOptimizer makes more sense as a name.

[FEATURE] Iterative tuning for catboost

Describe the solution you'd like
Utility function to tune catboost models iteratively

[BUG] Prerelease not working

Describe the bug
It always publish a release version

[Feature] Set default monitor converter

Is your feature request related to a problem? Please describe.

When using a monitor,

result = suggest_keras_models_by_sha(
    space.sample(16,0),
    plan=[(warmup+1.0,8),(2.0,4),(4.0,2),(8.0,1)],
    execution_engine_conf={"callback":True},
    monitor="rungs",
    top_n=2,
)

raises

TuneCompileError: rungs can't be converted to Monitor

because there is not default set on this line. Ideally, this is something that can be handled by default by all of the suggest_ functions.

Describe the solution you'd like

Right now, there is some manual code that needs to be added to get it working. This should just be the default behavior.

from typing import Optional
from tune import Monitor
from tune_notebook import (
    NotebookSimpleHist,
    NotebookSimpleRungs,
    NotebookSimpleTimeSeries,
    PrintBest,
)

def to_monitor(obj) -> Optional[Monitor]:
    if obj is None:
        return None
    if isinstance(obj, Monitor):
        return obj
    if isinstance(obj, str):
        if obj == "hist":
            return NotebookSimpleHist()
        if obj == "rungs":
            return NotebookSimpleRungs()
        if obj == "ts":
            return NotebookSimpleTimeSeries()
        if obj == "text":
            return PrintBest()
    raise NotImplementedError(obj)

TUNE_OBJECT_FACTORY.set_monitor_converter(to_monitor)

[BUG] hyperopt version 0.2.7 throws numpy error

Minimal Code To Reproduce

from sklearn.linear_model import LogisticRegression
from xgboost import XGBClassifier

from tune import Space, Grid, Rand, RandInt, Choice
from tune_sklearn import sk_space, suggest_sk_models

space = sum([
    sk_space(LogisticRegression),
    sk_space(XGBClassifier, n_estimators=Grid(5,10), max_depth=Grid(5,10), 
             learning_rate=Rand(0.01,0.99), eval_metric='mlogloss', n_jobs=1).sample(5,1)
])

result = suggest_sk_models(
    space,
    train, test,
    scoring="accuracy",
)

for r in result:
    print(r.metric, r.trial.keys, r.trial.params)

Describe the bug

This will throw an attribute error:

'numpy.random.mtrand.RandomState' object has no attribute 'integers'

This is because Optuna 0.2.7 (and I think 0.2.6) dropped the use of np.random.RandomState in place of np.random.Generator. More on this can be found on this. Downgrading numpy does not work to get solve this.

Expected behavior
A clear and concise description of what you expected to happen.

Environment (please complete the following information):

• Backend: native
• Backend version: fugue 0.6.5
• Python version: 3.7
• OS: linux

[FEATURE] Add Documents and comments

Describe the solution you'd like
Add all documents and comments in code

[FEATURE] Early stop for non iterative problems

Describe the solution you'd like
We can reuse the Judge class for iterative problems

1. If for certain criteria (for example, no improvement in certain period), judge can stop accepting new result
2. when judge stops, running trials should stop, so we need to change the execution design on worker side, it should be multiple process, one is watcher, regularly query the judge if it should quit

[FEATURE] Initialize and finalize in monitoring

Describe the solution you'd like
Monitoring although used in DAG, should have hooks to initialize and finalize

[FEATURE] Make HyperoptRunner extendable

Describe the solution you'd like
The class should be able to take extra arguments to configure hyperopt fmin, for example it needs to be able to configure early stop

[BUG] Dataset shuffle is not perfect

Describe the bug
It only shuffles within partition, we need a complete shuffle, but the implementation is not working in dask, we need to fix the issue of per_row that does not work well for dask dataframe.

[BUG] Bayesian optimizations do not respect min_better

Minimal Code To Reproduce

from tune import suggest_for_noniterative_objective, Space, Grid, Rand, noniterative_objective

@noniterative_objective(min_better=False)
def objective(x:float, y:float) -> float:
    return x*x + y*y

s2 = Space(x=Rand(-5,5),y=Rand(-3,3))

suggest_for_noniterative_objective(
    objective, s2,
    local_optimizer="hyperopt",
)

Describe the bug
It still tries to find the smallest value, Optuna has the same problem.

The problems are at:

tune/tune_optuna/optimizer.py

Line 44 in 173cc11

report = func.run(trial.with_params(params))

tune/tune_hyperopt/optimizer.py

Line 52 in 173cc11

report = func.run(trial.with_params(params))

We should use safe_run at these two places

Expected behavior
They should find the largest metric

Environment (please complete the following information):

• Backend: pandas/dask/ray?
• Backend version:
• Python version:
• OS: linux/windows

[FEATURE] Local optimizer should take stochastic expressions in nested structures

Is your feature request related to a problem? Please describe.

For this simplest example, p11 is a random expression but not on the top level.

dict(
    p1=dict(p11=Rand(0,1))
)

We need our local optimizers, currently optuna and hyperopt to be able to handle it.

[FEATURE] Iterative tuning for lgbm

Describe the solution you'd like
Utility function to tune light gbm models iteratively

[FEATURE] Integrate Wandb

Describe the solution you'd like
Integrate Wandb, something like

from tune import suggest_for_noniterative_objective, Space, Grid

def objective(x:float, y:float, z:float) -> float:
    return x*x + y*y + z*z

s1 = Space(x=Grid(2,3),y=Grid(1,3,4), z=Grid(6,7))
s2 = Space(x=Rand(-5,5),y=Rand(-3,3), z=Rand(1,3))

suggest_for_noniterative_objective(
    objective, s1+s2,
    local_optimizer="hyperopt:10",
    logger="wandb:proj1",
)

[FEATURE] 100% test coverage

Describe the solution you'd like
We need 100% unit test coverage

[FEATURE] Optuna integration

Describe the solution you'd like
Integrate Optuna as an objective runner