kudkudak/r2-learner

Code supporting "On certain limitations of Recursive Representation Models" paper

On certain limitations of Recursive Representation Models

This repository contains code for "On certain limitations of Recursive Representation Models" paper. You can also find here implementations of models from Learning with Recursive Perceptual Representations and Learning deep representations via extreme learning machines.

R2SVMLearner and R2ELMLearner

If you are primarly interested in implementations of those models you need only r2.py file, where you will also find simple tests. Implementation follows scikit-learn convention, but note there might be some problems with methods using set_params call. Bear in mind that this is just research implementation.

Reproducing results

For reproducing results you need to fit all the models. Change n_jobs parameter to speed up computation.

• Fit SVM, Linear SVM and SVM RBF

python scripts/fit_extern.py 

• Fit R2SVM and DrELM

python scripts/fit_r2.py

• Fit some models on tripled datasets

python scripts/fit_triple.py

• Fit randomized versions of R2M models

python scripts/fit_random.py

r2-learner's Issues

Use Python-ELM in R2ELM

https://github.com/dclambert/Python-ELM

GridSearchCV - saving intermediate results and models

We need GridSearchCV to save models that we learned, save intermediate results to restart in case of problems, etc. This should be simple, just steal scitkit-learn code and modify

One implementation
https://github.com/jennyyuejin/Kaggle/blob/master/CV_Utilities.py

Check if everything is correct

Run unit_tests from notebooks
Fix paths

Upgrade grid search function.

Run CV few times, report accuracy variance, return mean accuracy.

Test additional datasets #2

• glass - small many classes
• aloi - has many classes
• news20 - sparse
• segment

Please do it on fermi. Coding on fermi is easy if you use sshfs

2-quadratic-snakes dataset and donut & doughnut

Speed up fitting C in each layer

Right now it's way too slow.

Passing function R2ELM

Add random fiting to R2 + last + c=1

Fix best run image

Clearly it is not our original image where we had at least basic separation

Find difference between 2 publications

As I mentioned two implementations of first pub are worse then 2 publications of second.

Refactore grid search

Add regularization to ELM and test it

Extensively compare our model to Wojtek's imlementation

Fix all the randomness and compare data in every layer.

Fitting C in each layer - methods

Try k random C instead of k-fold GridSearchCV.

Test additional datasets

• satimage
• pendigits
• covtype
• mnist

Please do it on fermi. Coding on fermi is easy if you use sshfs

Add DBN to tests

https://github.com/rasmusbergpalm/DeepLearnToolbox

Octave :(

How to achieve those results on 2 class datasets?

Explore sigmoid and rbf alternative definition

Is it possible that author used different activation than elementwise sigmoid

x_j = sigmoid([x_j + (alpha *o_i * W_j)_j] w_j)

decision_function returns more dimensions then needed

Probable use wrapper we had, or use OneVsAll wrapper :) Your call

Prepare better tests.

• more data sets
• better result presentation
• better script
• mutliprocessing

Add tests to r2.py and cleanup code