mlr-org / mlr3tuning Goto Github PK
View Code? Open in Web Editor NEWHyperparameter optimization package of the mlr3 ecosystem
Home Page: https://mlr3tuning.mlr-org.com/
License: GNU Lesser General Public License v3.0
Hyperparameter optimization package of the mlr3 ecosystem
Home Page: https://mlr3tuning.mlr-org.com/
License: GNU Lesser General Public License v3.0
Something like this does not work at the moment due to the filtering of the tasks:
task = mlr3::mlr_tasks$get("iris")
learner = mlr3::mlr_learners$get("classif.rpart")
learner$param_vals = list(minsplit = 3)
resampling = mlr3::mlr_resamplings$get("cv")
resampling$param_vals = list(folds = 4L)
param_set = paradox::ParamSet$new(params = list(
paradox::ParamDbl$new("cp", lower = 0.001, upper = 0.1
)))
terminator = TerminatorEvaluations$new(5L)
ff = FitnessFunction$new(task, learner, resampling, param_set)
inner_tuner = TunerRandomSearch$new(ff, terminator)
outer = mlr3::mlr_resamplings$get("cv")
TunerNestedResampling$new(nested, outer)
#> Error: Stacking of nested resampling tuner is not supported.
I even don't know if we want this or stick to the error message?
Terminate tuning if a pre-specified performance is reached.
Tuners and Terminators should have slots $param_set
and $param_vals
so that we can tune them, too.
The fitness function could return the benchmark result from eval(design
that is then set individually for each new tuner.
and spawn all configs in parallel as default
document this properly
the public method seems to be undocumented
why is this not an active binding?
i like the explicit constrcution of the fitnessfunction. but can we can 1 bit of syntactic sugar, so we dont have to do that explicitly all the time?
This is (I guess) a pretty annoying issue. With deep = TRUE
the terminator of gs
is used within the hooks of gs1
but the terminator that should stop gs1
doesn't get updates. Therefore, tuning runs forever:
> learner = mlr3::mlr_learners$get("classif.rpart")
> learner$predict_type = "prob"
> resampling = mlr3::mlr_resamplings$get("holdout")
> measures = mlr3::mlr_measures$mget(c("auc", "mmce"))
> param_set = paradox::ParamSet$new(
+ params = list(
+ paradox::ParamDbl$new("cp", lower = 0.001, upper = 0.1),
+ paradox::ParamInt$new("minsplit", lower = 2, upper = 5)
+ )
+ )
>
> ff = FitnessFunction$new(
+ task = task,
+ learner = learner,
+ resampling = resampling,
+ measures = measures,
+ param_set = param_set,
+ ctrl = tune_control(store_prediction = TRUE) # for the exceptions
+ )
>
> terminator = TerminatorEvaluations$new(5)
> gs = TunerRandomSearch$new(ff, terminator)
>
> gs1 = gs$clone(deep = TRUE)
> gs1$tune()
INFO [mlr3] Benchmarking 5 experiments
INFO [mlr3] Running learner 'classif.rpart1' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart2' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart3' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart4' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart5' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Finished benchmark
INFO [mlr3] Benchmarking 5 experiments
INFO [mlr3] Running learner 'classif.rpart1' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart2' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart3' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart4' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart5' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Finished benchmark
INFO [mlr3] Benchmarking 5 experiments
INFO [mlr3] Running learner 'classif.rpart1' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart2' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart3' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart4' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart5' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Finished benchmark
INFO [mlr3] Benchmarking 5 experiments
INFO [mlr3] Running learner 'classif.rpart1' on task 'spam (iteration 1/1)' ...
INFO [mlr3] Running learner 'classif.rpart2' on task 'spam (iteration 1/1)' ...
^C
> gs$terminator
TerminatorEvaluations with 0 remaining evaluations
> gs1$terminator
TerminatorEvaluations with 5 remaining evaluations
The following does not error, but does not behave as it should:
library(mlr3)
library(mlr3tuning)
library(paradox)
# Define the ParamSet
ps = ParamSet$new(params = list(ParamDbl$new(id = "cp", lower = 0, upper = .8)))
lrn = mlr_learners$get("classif.rpart", param_vals = list(minsplit = 2))
terminator = TerminatorEvaluations$new(20)
=> We set an additional hardcoded minsplit
at = AutoTuner$new(lrn, "cv3", measures = "classif.acc", ps,
terminator, tuner = TunerRandomSearch, tuner_settings = list())
at$param_set$values
named list()
=> We seem to loose the param_vals here
at$train("iris")
at$param_set$values
$cp
[1] 0.3260657
this is simply the the BMR of the PerformanceEvaluator
create an option so that this is not stored by default, but settable
simply at the end of AutoTune$train store the bmr slot of the PE in an AT slot
I just saw code like this:
It looks horrible
TunerGenSA = R6Class("TunerGenSA",
inherit = Tuner,
public = list(
initialize = function(pe, terminator, ...) {
if (any(pe$param_set$storage_type != "numeric")) {
err_msg = "Parameter types needs to be numeric"
lg$error(err_msg)
stopf(err_msg)
}
# Default settings:
settings = list(smooth = FALSE, acceptance.param = -15,
simple.function = FALSE, temperature = 250)
super$initialize(id = "GenSA", pe = pe, terminator = terminator,
settings = insert_named(settings, list(...)))
}
),
private = list(
tune_step = function() {
blackBoxFun = function (x, pe) {
# set measure
measure = pe$measures[[1L]]
hashes = pe$bmr$data$hash
# convert ParamSet to data.table
x = setDT(as.list(x))
pe$eval(x)
new_hash = setdiff(pe$bmr$data$hash, hashes)
# calculate performance
perf = pe$bmr$resample_result(new_hash)$aggregate(measure)
if (measure$minimize) perf else -perf
return (perf)
}
self$GenSA_res = GenSA(fn = blackBoxFun, lower = self$pe$param_set$lower, upper = self$pe$param_set$upper,
control = self$settings, pe = self$pe)
}
)
)
Use nelder mead from base::optim
a) simply for logging
b) so we support multicrit optimization
note that this needs to be properly doced in the interface
With the default being store.tune = FALSE
.
the grid should be searched in batches, and we should shuffle the designpoints randomly before we do that
and review/optimize it
this was really missing in mlr
the user should be allowed to define, in each tuning step
the member function aggregate
should also include parameter values:
d = as.data.table(bmr$data)
d$pars = map(...)
unnest(d, "pars")
Edit: Redirected from mlr
It would be nice to have an optional progress bar that will show, either in console or in a separate window, the % done.
Thanks!
learners = mlr3misc::imap(design$transpose(), function(xt, i) {
learner = self$learner$clone()
learner$param_set$values = insert_named(learner$param_set$values, xt)
learner$id = paste0(learner$id, n_evals + i)
return(learner)
})
please do NOT change stuff like this internally
the ID should stay as it is
and please unit tests that this does not happen
fitness_function = FitnessFunction$new(
task = mlr_tasks$get("iris"),
learner = mlr_learners$get("classif.ranger"),
resampling = mlr_resamplings$get("cv"),
param_set = ParamSet$new(params = list(
ParamInt$new("mtry", lower = 1, upper = 4)
)))
terminator_1 = TerminatorEvaluations$new(max_evaluations = 1)
tuner_1 = TunerRandomSearch$new(ff = fitness_function, terminator = terminator_1, batch_size = 1)
tuner_1$tune()
tuner_1$aggregated()
# Error in rbindlist(x[[col]], fill = TRUE, use.names = TRUE) :
# Item 1 of input is not a data.frame, data.table or list
Maybe this is a common enough case
ff <- PerformanceEvaluator$new(task, lrn, cv, ps)
tuner <- TunerRandomSearch$new(ff, TerminatorEvaluations$new(10))
result <- tuner$tune()$tune_result()
that it would be worth it to define a shortcut, e.g.
result <- TuneRandom(task, lrn, cv, ps, 10)
Stop tuning after a pre-specified amount of time.
I assume that it would be nice to allow the Terminator to filter out configurations during update_start()
so that we can effectively control the termination inside batches of jobs (AFAIK this is not possible at the moment). E.g., for something like TerminationWalltime
we could just throw away all new configurations after the time budget for the optimization is exhausted. TerminationThreshold
could do the same as soon as a suitable configuration is found in the $experiments
.
Similarly, update_end()
could remove experiments (or change their performance values to the worst possible value or NA) for experiments which were performed illegally.
I guess something like this would be nice:
xs = self$terminator$update_prepare(ff, xs) # may remove points from xs
self$ff$eval_vectorized(xs)
self$terminator$update_finalize(ff) # may set performances to NA
Additionally, while passing the FF is not the worst idea, maybe we can break it down to something simpler? I find it confusing that the terminator operates on the class where it is stored.
This terminator should measure the improvement and stops after, let's say, 5 iterations if no increase in performance is measured.
At the moment the tuning method is passed as factory:
at = AutoTuner$new(learner, resampling, param_set, terminator, tuner = TunerGridSearch,
tuner_settings = list(resolution = 10L))
This is not very nice. We should pass an instantiated tuner object. The problem here is, that the tuner then already contains the task and it cannot be set by the autotuner which is required for nested resampling. Suggestion: The tune()
member function of tuner gets a task for the tuning. This would be more consistent with train(task)
and therefore much easier to use within the auto tuner.
What do you think about that?
a) the optimal obtained configuration
b) the tuning path leading there
The auto
in the naming is misleading, it sounds like there happens something magic. Maybe we should think about renaming the learner.
Do we need it? Can we just return self
invisibly?
We have to replace the resampling object of the fitness function with an ResamplingCustom
object to correctly evaluate the performance. This is not possible at the moment. See mlr-org/mlr3#108
At the moment we just use a holdout to see that it works in general:
# We want this:
resampling_temp = ResamplingCustom$new()
resampling_temp$instantiate(self$ff$task, train_sets = list(train_set_temp))
# But doint that:
resampling_temp = ResamplingHoldout$new()
self$ff$resampling = resampling_temp
# To use the fitness function functionality:
self$ff$eval(tuner_temp$tune_result()$param_vals)
I also think that we should keep this structure and use eval
of the fitness function since this handles the registration in the benchmark result etc. automatically.
many learners have a "sequence" parameter so you can get full trace from a single training run.
e.g. ntree, boosting rounds, or the "s" param in glmnet
this is should be supported.
basically we need to allow to "mark up" one hyperparam in the set, with a tag.
and then maybe not have a new predict_trace function but handle this in the normal predict
From the old mlr help page:
"Returns the optimal hyperparameters and optimization path after training."
Could inherit the options of getTuneResultOptPath()
.
The terminator should call start end end by itself
We need this. Do also describe how nested resampling can be done with the AutoTuner.
Parameter transformation should be executed from the fitness function, not the tuner.
it currently has none, but has a "state", the number of performed iterations, and so own
what does mlr3 here?
and should the PerformanceEvaluator have a printer?
This needs to be done. Test setup could be:
error_train
and error_predict
Add a section to the tuner vignette on how to see what really happens during the tuning.
Is this just good for parallelization or correct stopping?
We should have just one eval method that takes a data.table of parameter values and evaluates the fitness function. Should be more consistent with paradox.
I want to tune over the following Parameter Space:
param_set = paradox::ParamSet$new(
params = list(
paradox::ParamDbl$new("classif.svm.cost", lower = 0.1, upper = 1),
paradox::ParamInt$new("pca.rank.", lower = 1, upper = 3)
)
)
calling
pe = PerformanceEvaluator$new(iris, learner, resampling, param_set)
terminator = TerminatorEvaluations$new(30)
tuner = TunerGridSearch$new(pe, terminator)$tune()
I would now assume that it searches through pca.rank
1,2,3 and cost
0.1, 0.2, 0.3, ..., 1
,
as this would evaluate to 30 evaluations.
Instead I get WARN [11:38:30.936] Set number of maximal evaluations to 21 to avoid multiple computation of the same grid.
I.e. it seems to somehow compute a weird resolution from the evaluations?
Is it good to have the word "function" in the name of that class? This might inflict that the created object might be a function but in fact its a class.
I see upcoming confusion.
Why not something similar to mlr like TuneWrapper
?
Explain what the fitness function is, do also mention paradox, what it is, and link to the vignette
FitnessFunction
Temrinator
works on the GenSA
exampleTunerGenSA
classThis shouldn't be too hard to automatically test every Tuner with each Terminator. Maybe there is something that can be copied or used from mlr3learners?
eval
is the key to write own tuner. Therefore, it should be clear how eval
inserts entries to the BenchmarkResult
and how to access the results.
thankfully mlr3 might simply solve this for us, see here mlr-org/mlr3#75
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.