Random forests methodologies for :
Libraries we use :
- Ranger (Wright and Ziegler 2015) : we use our own fork and have tuned forests to do “online”[1] computations (Growing trees AND making predictions in the same pass, which removes the need of in-memory storage of the whole forest)[2].
- Eigen3 (Guennebaud, Jacob, and others 2010)
As a mention, we use our own implementation of LDA and PLS from (Friedman, Hastie, and Tibshirani 2001, 1:81, 114).
There is one set of binaries, which contains a Macos/Linux/Windows (x64 only) binary for each platform. There are available within the “Releases” tab, under “Assets” section (unfold it to see the list).
This is pure command line binary, and they are no prerequisites or library dependencies in order to run it. Just download them and launch them from your terminal software of choice. The usual caveats with command line executable apply there : if you’re not proficient with the command line interface of your platform, please learn some basics or ask someone who might help you in those matters.
As a note, we may add a graphical interface in a near future.
- ABC Random Forest - Model choice or parameter estimation command line options
Usage:
abcranger [OPTION...]
-h, --header arg Header file (default: headerRF.txt)
-r, --reftable arg Reftable file (default: reftableRF.bin)
-b, --statobs arg Statobs file (default: statobsRF.txt)
-o, --output arg Prefix output (modelchoice_out or estimparam_out by
default)
-n, --nref arg Number of samples, 0 means all (default: 0)
-m, --minnodesize arg Minimal node size. 0 means 1 for classification or
5 for regression (default: 0)
-t, --ntree arg Number of trees (default: 500)
-j, --threads arg Number of threads, 0 means all (default: 0)
-s, --seed arg Seed, generated by default (default: 0)
-c, --noisecolumns arg Number of noise columns (default: 5)
--nolinear Disable LDA for model choice or PLS for parameter
estimation
--chosenscen arg Chosen scenario (mandatory for parameter
estimation)
--ntest arg number of oob testing samples (mandatory for
parameter estimation)
--parameter arg name of the parameter of interest (mandatory for
parameter estimation)
--help Print help
- If you provide
--chosenscen
,--parameter
and--ntest
, parameter estimation mode is selected. - Otherwise by default it’s model choice mode.
- Linear additions are LDA for model choice and PLS for parameter estimation, “–nolinear” options disables them in both case.
Example :
abcranger -t 10000 -j 8
Header, reftable and statobs files should be in the current directory.
Four files are created :
modelchoice_out.ooberror
: OOB Error rate vs number of trees (line number is the number of trees)modelchoice_out.importance
: variables importance (sorted)modelchoice_out.predictions
: votes, prediction and posterior error ratemodelchoice_out.confusion
: OOB Confusion matrix of the classifier
The Pls components are selected within at least 99% of the maximum explained variance of the output.
where
is the
scored by the pls for the
th component. We take only the first
components, we stop when :
We can easily prove than
is superior or equal to
:
In practice, we find
close enough to
.
Computing the whole OOB set for weights predictions (Raynal et al.
2016), is very costly, memory and cpu-wise, so we
advise to compute them for only choose a subset of size ntest
.
Example (working with the dataset in test/data
) :
abcranger -t 1000 -j 8 --parameter ra --chosenscen 1 --ntest 50
Header, reftable and statobs files should be in the current directory.
Five files (or seven if pls activated) are created :
estimparam_out.ooberror
: OOB MSE rate vs number of trees (line number is the number of trees)estimparam_out.importance
: variables importance (sorted)estimparam_out.predictions
: expectation, variance and 0.05, 0.5, 0.95 quantile for predictionestimparam_out.predweights
: csv of the value/weights pairs of the prediction (for density plot)estimparam_out.teststats
: various statistics on test (MSE, NMSE, NMAE etc.)
if pls enabled :
estimparam_out.plsvar
: variance explained by number of componentsestimparam_out.plsweights
: variable weight in the first component (sorted by absolute value)
- Integrate hdf5 (or exdir? msgpack?) routines to save/load
reftables/observed stats with associated metadata
- Provide R code to save/load the data
- Provide Python code to save/load the data
- Merge the two methodologies in a single executable with the (almost) the same options
- (Optional) Possibly move to another options parser (CLI?)
- R package
- Python package
- Code documentation
- Document the build
- Fix travis build. Currently the vcpkg download of eigen3 head is broken.
- osX travis build
- Appveyor win32 build
- methodologies parameters auto-tuning
- auto-discovering the optimal number of trees by monitoring OOB error
- auto-limiting number of threads by available memory
- Streamline the two methodologies (model choice and then parameters estimation)
- Write our own tree/rf implementation with better storage efficiency than ranger
- Make functional tests for the two methodologies
Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. 2001. The Elements of Statistical Learning. Vol. 1. 10. Springer series in statistics New York, NY, USA:
Guennebaud, Gaël, Benoît Jacob, and others. 2010. “Eigen V3.” http://eigen.tuxfamily.org.
Pudlo, Pierre, Jean-Michel Marin, Arnaud Estoup, Jean-Marie Cornuet, Mathieu Gautier, and Christian P Robert. 2015. “Reliable Abc Model Choice via Random Forests.” Bioinformatics 32 (6): 859–66.
Raynal, Louis, Jean-Michel Marin, Pierre Pudlo, Mathieu Ribatet, Christian P Robert, and Arnaud Estoup. 2016. “ABC Random Forests for Bayesian Parameter Inference.” arXiv Preprint arXiv:1605.05537.
Wright, Marvin N, and Andreas Ziegler. 2015. “Ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R.” arXiv Preprint arXiv:1508.04409.
-
The term “online” there and in the code has not the usual meaning it has, as coined in “online machine learning”. We still need the entire training data set at once. Our implementation is an “online” one not by the sequential order of the input data, but by the sequential order of computation of the trees in random forests, sequentially computed and then discarded.
-
We only use the C++ Core of ranger, which is under MIT License, same as ours.