maximilian-gelbrecht/parameterschedulers.jl

Common hyperparameter scheduling for ML

ParameterSchedulers

ParameterSchedulers.jl provides common machine learning (ML) schedulers for hyper-parameters. Though this package is framework agnostic, a convenient interface for pairing schedules with Flux.jl optimizers is available. Using this package with Flux is as simple as:

using Flux, ParameterSchedulers
using ParameterSchedulers: Scheduler

opt = Scheduler(Exp(λ = 1e-2, γ = 0.8), Momentum())

Available Schedules

This is a table of the common schedules implemented, but ParameterSchedulers provides utilities for creating more exotic schedules as well. The higher order schedules should make it so that you will rarely need to write a schedule from scratch.

You can read this paper for more information on the schedules below.

{cell=table, display=false, output=false, results=false}

using UnicodePlots, ParameterSchedulers

Schedule	Description	Type	Example
Step(;λ, γ, step_sizes)	Exponential decay by γ every step in step_sizes	Decay	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = Step(λ = 1.0, γ = 0.8, step_sizes = [2, 3, 2]) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
Exp(;λ, γ)	Exponential decay by γ every iteration	Decay	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = Exp(λ = 1.0, γ = 0.5) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
CosAnneal(;λ0, λ1, period)	Cosine annealing	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = CosAnneal(λ0 = 0.0, λ1 = 1.0, period = 4) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
Triangle(;λ0, λ1, period)	Triangle wave function	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = Triangle(λ0 = 0.0, λ1 = 1.0, period = 2) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
TriangleDecay2(;λ0, λ1, period)	Triangle wave function with half the amplitude every period	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = TriangleDecay2(λ0 = 0.0, λ1 = 1.0, period = 2) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
TriangleExp(;λ0, λ1, period, γ)	Triangle wave function with exponential amplitude decay at rate γ	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = TriangleExp(λ0 = 0.0, λ1 = 1.0, period = 2, γ = 0.8) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
Poly(;λ, p, max_iter)	Polynomial decay at degree p	Decay	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = Poly(λ = 1.0, p = 2, max_iter = t[end]) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
Inv(;λ, γ, p)	Inverse decay at rate (1 + tγ)^p	Decay	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = Inv(λ = 1.0, p = 2, γ = 0.8) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
Sin(;λ0, λ1, period)	Sine function	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = Sin(λ0 = 0.0, λ1 = 1.0, period = 2) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
SinDecay2(;λ0, λ1, period)	Sine function with half the amplitude every period	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = SinDecay2(λ0 = 0.0, λ1 = 1.0, period = 2) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)
SinExp(;λ0, λ1, period)	Sine function with exponential amplitude decay at rate γ	Cyclic	{cell=table, display=false} using UnicodePlots, ParameterSchedulers t = 1:10 |> collect s = SinExp(λ0 = 0.0, λ1 = 1.0, period = 2, γ = 0.8) lineplot(t, s.(t); width = 15, height = 3, border = :ascii, labels = false)