arrayfire/arrayfire-ml

ArrayFire's Machine Learning Library.

arrayfire-ml's Issues

Linear Discriminant Analysis

• Requires SVD from arrayfire

Add Initializers

Currently Nodes are defined to have spreads:

            LinearNode(const int inputSize, const int outputSize,
                       float spread = 0.05,
                       const char *name="none") :

This needs to be refactored to provide an initializations class.
Examples of initializations are :

1. Le-Cun Normal
2. Glorot Uniform

Support Vector Machine

Extends classifier. Needs more thought.

Autodiff

Hi guys,

I've seen a lot of things in terms of ML frameworks, however for my work as a research I find autodiff very usefull (e.g. symbolic graph manipulation). On the other hand arrayfire seems like the perfect the backend for such framework, as it seemsly can be adopted for GPUs and so on (I'm still interested if there is any chance of adding somehow fallback to cuDNN where possible).

Since I've atempted such things, and however lack the experience of well written performance code, was wondering how open are you guys to such proposal and would anyone be interested in giving help for making better code generation to arrayfire.

I understand this is not your main goal, but I find arrayfire probably as best candidate for a computational backend, while autodiff gives a few orders of magnititude productivity. If I happen to this I do intend to do it in c++.

Example: Logistic and Softmax Regression

Specialization of Network class for:

• logistic regression
• softmax regression

Base classifier class

This class is inherited from all the classifier types. This should provide methods for training and then performing the classify operation.

TODO List for 0.1 release

Base Classes

• nn::Module
• autograd::Variable
• Solver
• nn::Loss

Autograd

• Broadcasting (sum, tile)
• Math (exp, pow, abs, sqrt, log)
• Binary (add, subtract, multiply, divide, min, max, logical operators)
• matrix multiplication
• Convolutions and strided Convolutions
• Indexing and assignment

Neural Network

• Linear
• Convolve
• Pooling (Min, Max, Average)
• Activations: Sigmoid, Tanh, ReLU and related
• Recurrent neural networks (RNN, LSTM, GRU)
• Various Losses
• Containers (Sequential)
• Initializers

Solvers / Optimizers

• SGD
• ADAM

Examples

Optimizers

• SGD
• Adam
• RMSProp

Remove bias from Weights.hpp

Weights are currently initializing bias'
This is not always desirable, eg LSTM:

            /* Standard LSTM Implementation
             * Reference: http://deeplearning.net/tutorial/lstm.html
             * Refernece: http://www.cs.toronto.edu/~graves/phd.pdf */
            Weights mWi, mUi, mIDiffs;   // input gate weights + recurrence
            Weights mWf, mUf, mFDiffs;   // forget gate weights + recurrence
            Weights mWc, mUc, mCDiffs;   // memory cell weights + recurrence
            Weights mWo, mUo, mODiffs;   // output gate weights + recurrence

Here, each node needs only needs 1 bias, i.e. 1 bias for input gate (as opposed to the 2 we will now have for mWi & mUi)
Either we should :

1. refactor the above file into 2 files or
2. we can provide a way to NOT initialize the bias in the CTOR

To me 1. makes more sense

HMM and Viterbi

arrayfire/arrayfire#376

Nearest Neighbor search

Use nearestNeighbour from arrayfire to implement this

Degraded performance for variable input size

I have extended arrayfire-ml with CuDNN bindings and was running benchmarks to compare with convnet benchmarks from https://github.com/soumith/convnet-benchmarks/.

The benchmarks are run in the following way

// Define the Network

// Benchmark code
for (int i = 0; i < ntimes; ++i) {
    af::sync();
    auto s = af::timer::start();
    input = <INPUT_INITIALIZATION USING AF::RANDU>
    auto out = model.forward(input);
    out.backward();
    af::sync();
    auto e = af::timer::stop(s);
    std::cout << std::setprecision(5) << e * 1000.0 << std::endl;
}

I was able to match the performance with torch7 CuDNN bindings if I make the input size always constant. However, if I pass random input between the sizes [lo, hi] the avg. performance if actually worse than sending input of size hi always.

You can notice the spike at regular intervals which is increasing the avg. time taken.

Note that all the buffers, arrays are initialized using af::array(..) constructor (No CudaMallocs used) and all Cudnn Operation are placed on Arrayfire's Cuda Stream on the device.

I was wondering if the spikes at regular intervals strike something to you. Could the continuous memory allocations (of different sizes) be optimized with Arrayfire memory manager ?

Thanks in advance !

OpenCL Error OSX Radeon HD6750

jason@tesla ~/p/a/build> ./test/Activations_opencl
in[0] [5 1 1 1]
-8.9262
32.2371
45.1800
-32.0635
-8.0176

libc++abi.dylib: terminating with uncaught exception of type af::exception: ArrayFire Exception(401): Double precision not supported for this device
In /var/lib/jenkins-slave/workspace/arrayfire-osx-graphics-installer/src/api/cpp/data.cpp:27
fish: Job 1, './test/Activations_opencl ' terminated by signal SIGABRT (Abort)

Naive Bayes Classifier

Extends classifier and has custom implementation

Automatic Differentiation

This would be cool: http://en.wikipedia.org/wiki/Automatic_differentiation
Theano uses this.

This should be a secondary objective. Initially (and for performance reasons) it will make sense to provide the derivatives of functions, however this feature will provide extensibility for later unknown model types.

Possible anachronism in CMake `target_include_directories`

When trying to compile, line 29 of the CMake file was giving me trouble. Changing the included item from ArrayFire::af to af solved the issue. I'm using CMake 3.10.2

See also:

1. https://cmake.org/cmake/help/latest/command/target_include_directories.html
2. https://gitlab.kitware.com/cmake/cmake/issues/15689

Deep Belief Networks

Specialization of Network class.

Multi Layer Perceptron

Specialization of Network class

Restricted Boltzman Machines and Autoencoders

Specializations of Network class.

• RBM
• Autoencoder
• StackedRBM
• StackedAutoencoder

Convolution Functions

The following functions and modules need to be implemented:

• Basic Convolutions: No strides, No pooling
• Strided Convolutions without pooling
• Strided Convolutions with pooling

Merging Strided Convolutions with pooling could potentially allow call wrap and unwrap to be only called once instead of twice.

RNN Models

Once we have an implementation of the Layer Class #17 , the Optimizer class and the DataSet class we can go about creating RNN flavors. There are 3 models that should be implemented:

• Vanilla RNN
• LSTM
• GRU

These will require the implementation of their derivatives and their forward prop values.
Certain details to consider:

• RNN's have a stack of weight matrices and bias' (not just 1 per Layer, thus the Layer needs to be general enough to handle this)
• The optimization needs to be handled via two methods:
  • RTRL (real time recurrent learning) &
  • BPTT (backprop through time)

To enable the above two methods of learning we should consider inheriting from Layer and implementing a Recurrent Layer.

Proposal for the initial API

This library will sit on top of arrayfire. The default implementation will be C++, however C wrappers will be added on top so that the library can be used from other languages.

The optimization function and distance metrics need to be abstract out so the user has a choice of switching them out whenever possible.

Base classes

• Classifier (Model fit ?)
• Neural Networks

Classifiers

• Perceptron
• Naive Bayes classifier
• Support Vector Machines (SVM)
• Logistic Regression
• Softmax Regression
• Multi Layer Perceptron
• Convolutional Neural Networks (CNN)
• Deep Belief Networks
• Nearest Neighbour Search (kNN)

Dimensionality Reduction

• Restricted Boltzman Machines (RBM)
• Autoencoders
• Principal Component Analysis (PCA)
• Linear Discriminant Analysis (LDA)
• Locality Sensitive Hashing (LSH)

Regression Analysis

• Linear Regression
• Genaralized Linear Regression (GLM)

Clustering

• k-Means

Optimization functions

Needs research

Example: Linear Regression

Specialization of Network class.

Indexing and assignment support in autograd

Static Activations Class

Should provide str2enum and then return a function ptr or something similar.
This will be helpful for things like:

        private:

            Weights mWeights;
            Weights mDiffs;
            Activation mActivation;

        public:

            LinearNode(const int inputSize, const int outputSize,
                       std::string activation='tanh',
                       float spread = 0.05,
                       const char *name="none"):
                Node(1, &inputSize, 1, &outputSize, name),
                mWeights(inputSize, outputSize, spread),
                mActivation.get(activation),
                mDiffs()
{}

            ArrayVector forward(const ArrayVector &input){
                return {mActivation(af::matmul(mWeights.getWeights(), input[0])) +
                        af::tile(mWeights.getBias(), 1, input[0].dims(1))};

k-means

Locality Sensitive Hashing

Principle Component Analysis

• Requires eigen value decomposition from arrayfire

Is ML project dead?

Didn't see any updates in a while, so wondering if ArrayFire ML project is still ongoing or abandoned.

Ideally would be great to have a "drop in" replacement for cuDNN used by Torch.

Example: Generalized Linear Regression

CUDNN support for Convolution Layers

Wraps perceptrons, convolutional and softmax layers

Other

• Integrate with cuDNN when possible
• Add a fallback version using cuDNN is not available.
• Investigate alternatives and analyze performance

Loss functions

• mean squared error
• mean absolute error
• binary cross entropy
• negative log likelyhood

Perceptron

Specialization of MultiLayerPerceptron