ibm/tsml.jl

A package for time series data processing, classification, clustering, and prediction.

Documentation	Build Status	Help

Stargazers over time

TSML (Timeseries Machine Learning)

---

TSML is a package for time series data processing, classification, clustering, and prediction. It combines ML libraries from Python's ScikitLearn (thru its complementary AutoMLPipeline package) and Julia MLs using a common API and allows seamless ensembling and integration of heterogenous ML libraries to create complex models for robust time-series prediction. The design/framework of this package is influenced heavily by Samuel Jenkins' Orchestra.jl and CombineML.jl packages. TSML is actively developed and tested in Julia 1.0 and above for Linux, MacOS, and Windows.

Links to TSML demo, tutorial, and published JuliaCon paper:

• TSML Binder Notebooks Live Demo
• Jupyter Notebook TSML Demo
• JuliaCon 2019 Proceedings Paper

Package Features

• Support for symbolic pipeline composition of transformers and learners
• TS data type clustering/classification for automatic data discovery
• TS aggregation based on date/time interval
• TS imputation based on symmetric Nearest Neighbors
• TS statistical metrics for data quality assessment
• TS ML wrapper with more than 100+ libraries from scikitlearn and julia
• TS date/value matrix conversion of 1-D TS using sliding windows for ML input
• Common API wrappers for ML libs from JuliaML, PyCall, and RCall
• Pipeline API allows high-level description of the processing workflow
• Specific cleaning/normalization workflow based on data type
• Automatic selection of optimised ML model
• Automatic segmentation of time-series data into matrix form for ML training and prediction
• Easily extensible architecture by using just two main interfaces: fit and transform
• Meta-ensembles for robust prediction
• Support for threads and distributed computation for scalability, and speed

Installation

TSML is in the Julia Official package registry. The latest release can be installed at the Julia prompt using Julia's package management which is triggered by pressing ] at the Julia prompt:

julia> ]
(v1.1) pkg> add TSML

Or, equivalently, via the Pkg API:

julia> using Pkg
julia> Pkg.add("TSML")

Motivations

Over the past years, the industrial sector has seen many innovations brought about by automation. Inherent in this automation is the installation of sensor networks for status monitoring and data collection. One of the major challenges in these data-rich environments is how to extract and exploit information from these large volume of data to detect anomalies, discover patterns to reduce downtimes and manufacturing errors, reduce energy usage, etc.

To address these issues, we developed TSML package. It leverages AI and ML libraries from ScikitLearn and Julia as building blocks in processing huge amount of industrial times series data. It has the following characteristics described below.

Main Workflow

The package assumes a two-column input composed of Dates and Values. The first part of the workflow aggregates values based on the specified date/time interval which minimizes occurrence of missing values and noise. The aggregated data is then left-joined to the complete sequence of dates in a specified date/time interval. Remaining missing values are replaced by k nearest neighbors where k is the symmetric distance from the location of missing value. This approach can be called several times until there are no more missing values.

TSML uses a pipeline of filters and transformers which iteratively calls the fit! and transform! families of functions relying on multiple dispatch to select the correct algorithm from the steps outlined above.

TSML supports transforming time series data into matrix form for ML training and prediction. Dateifier filter extracts the date features and convert the values into matrix form parameterized by the size and stride of the sliding window representing the dimension of the input for ML training and prediction. Similar workflow is done by the Matrifier filter to convert the time series values into matrix form.

The final part combines the dates matrix with the values matrix to become input of the ML with the output representing the values of the time periods to be predicted ahead of time.

Machine learning functions in TSML are wrappers to the corresponding Scikit-learn and native Julia ML libraries. There are more than hundred classifiers and regression functions available using a common API. In order to access these Scikit-learn wrappers, one should load the related package called AutoMLPipeline.

Below are examples of the Pipeline workflow.

• Load TSML and setup filters/transformers

# Setup source data and filters to aggregate and impute hourly
using TSML 

fname        = joinpath(dirname(pathof(TSML)),"../data/testdata.csv")
csvread      = CSVDateValReader(Dict(:filename=>fname,:dateformat=>"dd/mm/yyyy HH:MM"))
aggregate    = DateValgator(Dict(:dateinterval=>Dates.Hour(1)))   # aggregator
impute       = DateValNNer(Dict(:dateinterval=>Dates.Hour(1)))    # imputer
chkstats     = Statifier(Dict(:processmissing=>true))             # get statistics
normtonic    = Monotonicer(Dict()) # normalize monotonic data
chkoutlier   = Outliernicer(Dict(:dateinterval => Dates.Hour(1))) # normalize outliers

• Pipeline to load csv data

pipexpr = csvread
data    = fit_transform!(pipexpr)
first(data,5)

5×2 DataFrame
│ Row │ Date                │ Value   │
│     │ DateTime            │ Float64 │
├─────┼─────────────────────┼─────────┤
│ 1   │ 2014-01-01T00:06:00 │ 10.0    │
│ 2   │ 2014-01-01T00:18:00 │ 10.0    │
│ 3   │ 2014-01-01T00:29:00 │ 10.0    │
│ 4   │ 2014-01-01T00:40:00 │ 9.9     │
│ 5   │ 2014-01-01T00:51:00 │ 9.9     │

• Pipeline to aggregate and check statistics

pipexpr = csvread |> aggregate |> chkstats
stats   = fit_transform!(pipexpr)

1×26 DataFrame. Omitted printing of 19 columns
│ Row │ tstart              │ tend                │ sfreq    │ count │ max     │ min     │ median  │
│     │ DateTime            │ DateTime            │ Float64  │ Int64 │ Float64 │ Float64 │ Float64 │
├─────┼─────────────────────┼─────────────────────┼──────────┼───────┼─────────┼─────────┼─────────┤
│ 1   │ 2014-01-01T00:00:00 │ 2015-01-01T00:00:00 │ 0.999886 │ 3830  │ 18.8    │ 8.5     │ 10.35   │

Note: fit_transform! is equivalent to calling in sequence fit! and transform! functions.

• Pipeline to aggregate, impute, and check stats

pipexpr = csvread |> aggregate |> impute |> chkstats
stats2  = fit_transform!(pipexpr)

1×26 DataFrame. Omitted printing of 19 columns
│ Row │ tstart              │ tend                │ sfreq    │ count │ max     │ min     │ median  │
│     │ DateTime            │ DateTime            │ Float64  │ Int64 │ Float64 │ Float64 │ Float64 │
├─────┼─────────────────────┼─────────────────────┼──────────┼───────┼─────────┼─────────┼─────────┤
│ 1   │ 2014-01-01T00:00:00 │ 2015-01-01T00:00:00 │ 0.999886 │ 8761  │ 18.8    │ 8.5     │ 10.0    │

• Pipeline to aggregate, impute, and normalize monotonic data

pipexpr = csvread |> aggregate |> impute |> normtonic  
fit_transform!(pipexpr)

8761×2 DataFrame
│ Row  │ Date                │ Value    │
│      │ DateTime            │ Float64? │
├──────┼─────────────────────┼──────────┤
│ 1    │ 2014-01-01T00:00:00 │ 10.0     │
│ 2    │ 2014-01-01T01:00:00 │ 9.9      │
│ 3    │ 2014-01-01T02:00:00 │ 10.0     │
│ 4    │ 2014-01-01T03:00:00 │ 10.0     │
│ 5    │ 2014-01-01T04:00:00 │ 10.0     │
│ 6    │ 2014-01-01T05:00:00 │ 10.0     │
│ 7    │ 2014-01-01T06:00:00 │ 10.0     │
⋮

• Transforming timeseries data into matrix form for ML Modeling

# create artificial timeseries data
datets  = DateTime(2018,1,1):Dates.Day(1):DateTime(2019,1,31) |> collect
valuets = rand(1:100,length(datets))
ts      = DataFrame(Date=datets,Value=valuets)
@show first(ts,5);

5×2 DataFrame
│ Row │ Date                │ Value │
│     │ DateTime            │ Int64 │
├─────┼─────────────────────┼───────┤
│ 1   │ 2018-01-01T00:00:00 │ 56    │
│ 2   │ 2018-01-02T00:00:00 │ 93    │
│ 3   │ 2018-01-03T00:00:00 │ 40    │
│ 4   │ 2018-01-04T00:00:00 │ 15    │
│ 5   │ 2018-01-05T00:00:00 │ 78    │

# Pipeline to concatinate matrified value and date series
args     = Dict(:ahead => 24,:size => 24,:stride => 5)
datemtr  = Dateifier(args)
valuemtr = Matrifier(args)
ppl      = datemtr + valuemtr
dateval  = fit_transform!(ppl,ts)
first(dateval,5)

5×33 DataFrame. Omitted printing of 21 columns
│ Row │ year  │ month │ day   │ hour  │ week  │ dow   │ doq   │ qoy   │ x1    │ x2    │ x3    │ x4    │
│     │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │ Int64 │
├─────┼───────┼───────┼───────┼───────┼───────┼───────┼───────┼───────┼───────┼───────┼───────┼───────┤
│ 1   │ 2019  │ 1     │ 7     │ 0     │ 2     │ 1     │ 7     │ 1     │ 94    │ 97    │ 18    │ 76    │
│ 2   │ 2019  │ 1     │ 2     │ 0     │ 1     │ 3     │ 2     │ 1     │ 99    │ 93    │ 65    │ 68    │
│ 3   │ 2018  │ 12    │ 28    │ 0     │ 52    │ 5     │ 89    │ 4     │ 88    │ 8     │ 59    │ 1     │
│ 4   │ 2018  │ 12    │ 23    │ 0     │ 51    │ 7     │ 84    │ 4     │ 76    │ 5     │ 6     │ 92    │
│ 5   │ 2018  │ 12    │ 18    │ 0     │ 51    │ 2     │ 79    │ 4     │ 6     │ 54    │ 66    │ 72    │

• ML Modeling and Prediction

We can use the matrified dateval as input features for prediction/classication. Let's create a dummy response consisting of yes or no and use Random Forest to learn the mapping. More examples of ML modeling can be found in TSML's complementary packages: AutoMLPipeline and AMLPipelineBase.

target        = rand(["yes","no"],nrow(dateval))
rf            = RandomForest()
accuracy(x,y) = score(:accuracy,x,y)
crossvalidate(rf,dateval,target,accuracy)

fold: 1, 14.285714285714285
fold: 2, 57.14285714285714
fold: 3, 71.42857142857143
fold: 4, 85.71428571428571
fold: 5, 57.14285714285714
fold: 6, 57.14285714285714
fold: 7, 57.14285714285714
fold: 8, 71.42857142857143
fold: 9, 42.857142857142854
fold: 10, 71.42857142857143
(mean = 58.57142857142857, std = 19.57600456294711, folds = 10)

Extending TSML

If you want to add your own filter or transformer or learner, take note that filters and transformers process the input features but ignores the output argument. On the other hand, learners process both their input and output arguments during fit! while transform! expects one input argument in all cases.

The first step is to import the abstract types and define your own mutable structure as subtype of either Learner or Transformer. Next is to import the fit! and transform! functions so that you can overload them. Also, you must load the DataFrames package because it is the main format for data processing. Finally, implement your own fit and transform and export them.

  using DataFrames
  using TSML.AbsTypes

  # import functions for overloading
  import TSML.AbsTypes: fit!, transform!

  # export the new definitions for dynamic dispatch
  export fit!, transform!, MyFilter

  # define your filter structure
  mutable struct MyFilter <: Transformer
    name::String
    model::Dict
    args::Dict
    function MyFilter(args::Dict())
        ....
    end
  end

# define your fit! function.
  function fit!(fl::MyFilter, inputfeatures::DataFrame, target::Vector=Vector())
       ....
  end

  #define your transform! function
  function transform!(fl::MyFilter, inputfeatures::DataFrame)::DataFrame
       ....
  end

Remember that the main format to exchange data is dataframe which requires transform! output to return a dataframe. The features as input for fit! and transform! shall be in dataframe format too. This is necessary so that the pipeline passes the dataframe format consistently to its corresponding filters or transformers or learners. Once you have create this transformer, you can use plug is as part of the pipeline element together with the other learners and transformers.

Feature Requests and Contributions

We welcome contributions, feature requests, and suggestions. Here is the link to open an issue for any problems you encounter. If you want to contribute, please follow the guidelines in contributors page.

Help usage

Usage questions can be posted in:

• Julia Community
• Gitter TSML Community
• Julia Discourse forum

tsml.jl's Issues

Parameterise aggregator function for DateValgator and DateValNNer

currently, implementation uses median. add maximum, minimum, and mean for possible usage in the future.

Implement cross-validation performance evaluation for classification

Add clustering filter

create wrapper to use clustering from Clustering.jl

Disable plotter trigger of plot window during testing

Create a CLI wrapper with Pre-compiled TSML

To let TSML be called from command line, create a docker image with precompiled TSML accepting the following arguments: source location+filename, dateformat, aggregate time interval, targetlocation+outputname

refactoring with reexport

@JuliaRegistrator register()

Precompile TSML using userimg.jl for faster execution

Precompile TSML into userimg.jl for faster loading and execution.

REST API support

TSML as a service

refactor and bug fixes - v2.3.7

@JuliaRegistrator register()

Monotonic TS processing

Detect if data is monotonic and transform it into delta form using diff function. Use the immediate neighbor to insert the first element.

Website link from the generated docs by Documenter.jl

Create a docker with options using ArgParse

• use ArgParse.jl to get options and defaults

Refactor TSML for simplified operations and leaner modules

Currently, external libs such as scikitlearn and caret are integrated into TSML which makes loading it slower and precompilation not possible due to binary deps. The solution is to separate these external libs into separate package called TSMLExtra and make them optional. Target is to make TSML be just reliant on pure Julia codes and lib including ML libs.

Create PDF documentation aside from HTML

Implement threading for Julia1.3

new patch version 2.3.8

@JuliaRegistrator register()

Default TS cleaning/normalization pipeline as one function for CLI

Create a higher-level function containing the default workflow to process data consisting of following steps:

1. csvfile reading
2. monotonic processing
3. Aggregating based on a given time-interval
4. Imputing with KNN
5. Get statistics

Make all filter return the data during transform

• all filters should always return the TS even if it is used to save in the file or plot it.

Implement Meta-ensembles

Meta-ensemble types to be implemented:

1. Voting - use mean/mode for regression/classification
2. RF mean - use RF scoring
3. Boosting - use Adaboost ensembling workflow
4. Best - pick the best ML based of cross-validation

Streamline and Refactor external libs

@JuliaRegistrator register()

Error tagging new release

The URL of this package does not match that stored in METADATA.jl.
cc: @julia-tagbot[bot]

Time Series Classifier

Given subdirectories containing datasets:
training/
testing/
model/

datasets are named based on their datatype together with a sequence number. during training, the algorithm extracts stat features, append the datatype in filename and train it on RF. during prediction, similar workflow and apply RF to the features for predicting the datatype.

new version v2.3.4

@JuliaRegistrator register()

Implement error logging

Necessary to find out which files were not processed and the reasons.

Documentation of main workflow with Documenter

Create Plotter: Plotting filter for TSML

Create a plot filter so that results of aggregation, imputation, monotonic detection can be visualize for verification and reports.

Version 2.0.0 Release

@JuliaRegistrator register()

Integrate TSML with KITT using Job Queue

Assumptions:

There are three different queues corresponding to the major processing functions:

1. DataCleaning Queue
2. Modeling Queue
3. Plotting Queue

The DataCleaning Queue contains jobs with the following information:

1. location of source data and filename to use
2. location of target output and filename to use
3. location of the next major processing queue

TSML calls datadownloader to get data, process it, and push the results using datauploader to the location specified in the target. TSML marks the job done and push the job to the next processing queue. Datadownloader and datauploader are the executables written in RUST and used as interface between KITT and data processing system.

E2D will be pulling jobs from the modeling queue, process it, and push the output to the target location specified by the job.

Plotting queue can be optional depending on the needs.

Make the CSVwriter more generic instead of just restricting to 2-column Date-Value format

Currently, CSVwriter is strict about the dataframes saved. To support both Date-Value TS and Statistics summary, remove the restriction of 2-columns TS.

Outlierlizer: Normalize outliers

Implement outlier normalizer using nearest neighbor. Outliers are values below q1-IQR or above q3+IQR

v2.3.3 - standard scaler

@JuliaRegistrator register()

Dockerization and Deployment of TSML

Error tagging new release

The URL of this package does not match that stored in METADATA.jl.
cc: @julia-tagbot[bot]

Add documentation of outlier normalizer

dd docs to the notebook demo as well as to the documenter markdown.

Automatic classification of TS data type

Implement an algorithm for automatic classification of data type for specialized data cleaning/normalization workflow driven by the type and nature of data.

V2.3.0

@JuliaRegistrator register()

Error tagging new release

The URL of this package does not match that stored in METADATA.jl.
cc: @ppalmes

add more scalar stats

total missing
total samples
avg samples per hour
q0.1,q0.2,q0.8,q0.9

Activate Registrator for TSML v1.0.7

@JuliaRegistrator register()

Activate Registrator to register release version to Julia General Registry

@JuliaRegistrator register()

V2.3.1 - ensemble methods

@JuliaRegistrator register()

TSMLextra: Refactor extra filesystem and external libs

• TSML for CSV files only and pure Julia ML
• TSML extra for other filesystem
• TSML extra for scikitlearn and care libs

Purpose: Make TSML smaller, faster loading faster/precompilation.

Database/REST interface filter

Aside from csv, it is typical for companies to store their data in a database (relational, graph). We need to have a data abstraction filter that accesses data using REST protocol or send sql query over the web.

remove redundant modules with the help of Reexport.jl

new version 2.3.5

@JuliaRegistrator register()

DOIs for references

My understanding of the review instructions for the JuliaCon proceedings is that your references bib file should contain DOI identifiers, e.g. for the Julia paper by Bezanson et al.

Error tagging new release

The URL of this package does not match that stored in METADATA.jl.
cc: @julia-tagbot[bot]

v2.3.2 - fix typo error of instances variable in decision tree

@JuliaRegistrator register()

Setup plot for publication and webpage rendering

• setup canvas size, resolution, etc

Error tagging new release

The URL of this package does not match that stored in METADATA.jl.
cc: @ppalmes