Allow TSML.Outliernicers.Outliernicer range to be changed.

[rustyconover]

Allow TSML.Outliernicers.Outliernicer range to be changed.

[ppalmes]

thanks for the feedback. which do you think is practical? absolute range or still a factor of IQR?

[ppalmes]

is this ok? let scale=1.5, then range is (median - scale x iqr, median + scale x iqr)

scale will be passed as an argument.

[ppalmes]

as a temporary solution, you can overwrite the transform! function.

using TSML
using Dates
using DataFrames
using Statistics
using StatsBase: iqr, quantile, sample

import TSML.transform! ## to override transform!

function transform!(st::Outliernicer, features::DataFrame)
             features != DataFrame() || return DataFrame()
             ncol(features) == 2 || error("dataframe must have 2 columns: Date, Val")
             sum(names(features) .== ("Date","Value"))  == 2 || error("wrong column names")
             mfeatures=deepcopy(features)
             rvals = mfeatures.Value
             # compute the outlier range
             # setup to store both missing and numbers
             mvals = Array{Union{Missing,eltype(rvals)},1}(missing,length(rvals))
             mvals .= rvals
             crvals = skipmissing(rvals) # stat of non-missing
             miqr = iqr(crvals)
             m  = quantile(crvals,0.5) ##median
             scale = 1.5 ## scale factor of iqr
             lower=m-scale*miqr; upper=m+scale*miqr
             missindx = findall(x -> !ismissing(x) && (x > upper || x < lower),rvals)
             mvals[missindx] .= missing
             mfeatures.Value = mvals
             # use ValNNer to replace missings
             valnner = DateValNNer(st.args)
             fit!(valnner,mfeatures)
             resdf = transform!(valnner,mfeatures)
             resdf.Value = collect(skipmissing(resdf.Value))
             resdf
end

[rustyconover]

Personally I like using IQR but I'm far from an expert.

I really like this package because I've been using it to replace missing values in some time series I've been working with. It makes it so simple. @ppalmes Thank you for creating it!

The Readme.md mentions:

TS ML wrapper with more than 100+ libraries from caret, scikitlearn, and julia

Are these examples of using that wrapper?

Rusty

[ppalmes]

Thanks for the nice feedback!

The latest master branch now contains the option to pass the scale factor:

    Outliernicer(Dict(
       :dateinterval => Dates.Hour(1),
       :nnsize => 1,
       :missdirection => :symmetric,
       :scale => 1.25
    ))

I refactor TSML and placed the ML part to AutoMLPipeline (https://github.com/IBM/AutoMLPipeline.jl). The moment the time-series is matrified using Matrifier and Dateifier (https://ibm.github.io/TSML.jl/dev/man/valueproc/), you can use the AutoMLPipeline to do the machine learning using ScikitLearn. Caret is problematic to be added because I want to support Windows/Linus/Mac. I highly recommend you to try AutoMLPipeline.

[ppalmes]

it will use the scale factor: (median - scale * IQR, median + scale * IQR)

after all test passes, I will register a new version with these changes.

[ppalmes]

PR #166

[ppalmes]

I just registered a new version. you can update your TSML copy once it's accepted in General registry.

[rustyconover]

@ppalmes thank you! :)