dclambert/python-elm

Extreme Learning Machine implementation in Python

Python-ELM v0.3

---> ARCHIVED March 2021 <---

This is an implementation of the Extreme Learning Machine [1][2] in Python, based on scikit-learn.

From the abstract:

It is clear that the learning speed of feedforward neural networks is in general far slower than required and it has been a major bottleneck in their applications for past decades. Two key reasons behind may be: 1) the slow gradient- based learning algorithms are extensively used to train neural networks, and 2) all the parameters of the networks are tuned iteratively by using such learning algorithms. Unlike these traditional implementations, this paper proposes a new learning algorithm called extreme learning machine (ELM) for single- hidden layer feedforward neural networks (SLFNs) which ran- domly chooses the input weights and analytically determines the output weights of SLFNs. In theory, this algorithm tends to provide the best generalization performance at extremely fast learning speed. The experimental results based on real- world benchmarking function approximation and classification problems including large complex applications show that the new algorithm can produce best generalization performance in some cases and can learn much faster than traditional popular learning algorithms for feedforward neural networks.

It's a work in progress, so things can/might/will change.

David C. Lambert
dcl [at] panix [dot] com

Copyright © 2013
License: Simple BSD

Files

random_layer.py

Contains the RandomLayer, MLPRandomLayer, RBFRandomLayer and GRBFRandomLayer classes.

RandomLayer is a transformer that creates a feature mapping of the inputs that corresponds to a layer of hidden units with randomly generated components.

The transformed values are a specified function of input activations that are a weighted combination of dot product (multilayer perceptron) and distance (rbf) activations:

  input_activation = alpha * mlp_activation + (1-alpha) * rbf_activation

  mlp_activation(x) = dot(x, weights) + bias
  rbf_activation(x) = rbf_width * ||x - center||/radius

mlp_activation is multi-layer perceptron input activation

rbf_activation is radial basis function input activation

alpha and rbf_width are specified by the user

weights and biases are taken from normal distribution of mean 0 and sd of 1

centers are taken uniformly from the bounding hyperrectangle of the inputs, and

radius = max(||x-c||)/sqrt(n_centers*2)

(All random components can be supplied by the user by providing entries in the dictionary given as the user_components parameter.)

The input activation is transformed by a transfer function that defaults to numpy.tanh if not specified, but can be any callable that returns an array of the same shape as its argument (the input activation array, of shape [n_samples, n_hidden]).

Transfer functions provided are:

• sine
• tanh
• tribas
• inv_tribas
• sigmoid
• hardlim
• softlim
• gaussian
• multiquadric
• inv_multiquadric

MLPRandomLayer and RBFRandomLayer classes are just wrappers around the RandomLayer class, with the alpha mixing parameter set to 1.0 and 0.0 respectively (for 100% MLP input activation, or 100% RBF input activation)

The RandomLayer, MLPRandomLayer, RBFRandomLayer classes can take a callable user provided transfer function. See the docstrings and the example ipython notebook for details.

The GRBFRandomLayer implements the Generalized Radial Basis Function from [3]

elm.py

Contains the ELMRegressor, ELMClassifier, GenELMRegressor, and GenELMClassifier classes.

GenELMRegressor and GenELMClassifier both take *RandomLayer instances as part of their contructors, and an optional regressor (conforming to the sklearn API)for performing the fit (instead of the default linear fit using the pseudo inverse from scipy.pinv2). GenELMClassifier is little more than a wrapper around GenELMRegressor that binarizes the target array before performing a regression, then unbinarizes the prediction of the regressor to make its own predictions.

The ELMRegressor class is a wrapper around GenELMRegressor that uses a RandomLayer instance by default and exposes the RandomLayer parameters in the constructor. ELMClassifier is similar for classification.

plot_elm_comparison.py

A small demo (based on scikit-learn's plot_classifier_comparison) that shows the decision functions of a couple of different instantiations of the GenELMClassifier on three different datasets.

elm_notebook.py

An IPython notebook, illustrating several ways to use the *ELM* and *RandomLayer classes.

Requirements

Written using Python 2.7.3, numpy 1.6.1, scipy 0.10.1, scikit-learn 0.13.1 and ipython 0.12.1

References

[1] http://www.extreme-learning-machines.org

[2] G.-B. Huang, Q.-Y. Zhu and C.-K. Siew, "Extreme Learning Machine:
          Theory and Applications", Neurocomputing, vol. 70, pp. 489-501,
          2006.
          
[3] Fernandez-Navarro, et al, "MELM-GRBF: a modified version of the  
          extreme learning machine for generalized radial basis function  
          neural networks", Neurocomputing 74 (2011), 2502-2510

python-elm's Issues

Missing function atleast2d_or_csr() - scikit-learn 0.17.0

Hi

Thanks for the code.

The function atleast2d_or_csr() used in random_layer.py does not seem to be available any more with the new scikit-learn 0.17.0

I managed to work around this temporarily by defining a function by that name which simply returns the matrix without performing any checks.

def atleast2d_or_csr(X):
return X

I'm not sure what checks should be in there or what the equivalent function in the new scikit is though.

Hope this helps

Cheers

Srimal.

How to use it ?

Please tell me how can i use it for my datasets. what steps are require to implement it.

TypeError: unsupported operand type(s) for -: 'map' and 'map'

elmc.fit(x_train,y_train)
Traceback (most recent call last):
File "", line 1, in
File "/home/analytics/anaconda3/lib/python3.6/site-packages/sklearn_extensions/extreme_learning_machines/elm.py", line 596, in fit
super(ELMClassifier, self).fit(X, y_bin)
File "/home/analytics/anaconda3/lib/python3.6/site-packages/sklearn_extensions/extreme_learning_machines/elm.py", line 464, in fit
self.genelm_regressor.fit(X, y)
File "/home/analytics/anaconda3/lib/python3.6/site-packages/sklearn_extensions/extreme_learning_machines/elm.py", line 183, in fit
self.hidden_activations = self.hidden_layer.fit_transform(X)
File "/home/analytics/anaconda3/lib/python3.6/site-packages/sklearn/base.py", line 517, in fit_transform
return self.fit(X, **fit_params).transform(X)
File "/home/analytics/anaconda3/lib/python3.6/site-packages/sklearn_extensions/extreme_learning_machines/random_layer.py", line 108, in fit
self._generate_components(X)
File "/home/analytics/anaconda3/lib/python3.6/site-packages/sklearn_extensions/extreme_learning_machines/random_layer.py", line 360, in _generate_components
self._compute_centers(X, sp.issparse(X), rs)
File "/home/analytics/anaconda3/lib/python3.6/site-packages/sklearn_extensions/extreme_learning_machines/random_layer.py", line 322, in _compute_centers
spans = max_Xs - min_Xs
TypeError: unsupported operand type(s) for -: 'map' and 'map'

I get this error when fitting and not able to solve it. My x_train is csr matrix and y_train is ndarray.Any suggestions on this?

The ELM Scandal: 5 Easy Steps to Academic Fame

The “extreme learning machines (ELM)” are indeed worth working on, but they just shouldn’t be called “ELM”. With annotated PDF files at http://elmorigin.wix.com/originofelm , you can easily verify the following facts within 10 to 20 minutes:

1. The kernel (or constrained-optimization-based) version of ELM (ELM-Kernel, Huang 2012) is identical to kernel ridge regression (for regression and single-output classification, Saunders ICML 1998, as well as the LS-SVM with zero bias; for multiclass multi-output classification, An CVPR 2007).
2. ELM-SLFN (the single-layer feedforward network version of the ELM, Huang IJCNN 2004) is identical to the randomized neural network (RNN, with omission of bias, Schmidt 1992) and another simultaneous work, i.e., the random vector functional link (RVFL, with omission of direct input-output links, Pao 1994).
3. ELM-RBF (Huang ICARCV 2004) is identical to the randomized RBF neural network (Broomhead-Lowe 1988, with a performance-degrading randomization of RBF radii or impact factors).
4. In all three cases above, G.-B. Huang got his papers published after excluding a large volume of very closely related literature.
5. Hence, all 3 "ELM variants" have absolutely no technical originality, promote unethical research practices among researchers, and steal citations from original inventors.

Please forward this message to your contacts so that others can also study the materials presented at this website and take appropriate actions, if necessary.

ELM: The Sociological Phenomenon

Since the invention of the name “extreme learning machines (ELM)” in 2004, the number of papers and citations on the ELM has been increasing exponentially. How can this be imaginable for the ELM comprising of 3 decade-old algorithms published by authors other than the ELM inventor? This phenomenon would not have been possible without the support and participation of researchers on the fringes of machine learning. Some (unknowingly and a few knowingly) love the ELM for various reasons:

• Some authors love the ELM, because it is always easy to publish ELM papers in an ELM conference or an ELM special issue. For example, one can simply take a decade-old paper on a variant of RVFL, RBF or kernel ridge regression and re-publish it as a variant of the ELM, after paying a small price of adding 10s of citations on Huang’s “classic ELM papers”.

• A couple of editor-in-chiefs (EiCs) love the ELM and offer multiple special issues/invited papers, because the ELM conference & special issues will bring a flood of papers, many citations and therefore high impact factors to their low quality journals. The EiCs can claim to have faithfully worked within the peer-review system, i.e. the ELM submissions are all rigorously reviewed by ELM experts.

• A few technical leaders, e.g. some IEEE society officers, love the ELM, because it rejuvenates the community by bringing in more activities and subscriptions.

• A couple of funding agencies love the ELM, because they would rather fund a new sexy name, than any genuine research.

One may ask: how can something loved by so many be wrong?

A leading cause of the current Greek economic crisis was that a previous government showered its constituents with jobs and lucrative compensations, in order to gain their votes, thereby raising the debt to an unsustainable level. At that time, the government behavior was welcome by many, but led to severe consequences. Another example of popularity leading to a massive disaster can be found in WW II as Hitler was elected by popular votes.

The seemingly small price to pay in the case of the ELM is the diminished publishing ethics, which, in a long run, will fill the research literature with renamed junk, thereby making the research community and respected names, such as IEEE, Thomson Reuters, Springer and Elsevier, laughing stocks. Similar to that previous Greek government and its supporting constituents, the ELM inventor and his supporters are “borrowing” from the future of the entire research community for their present enjoyment! It is time to wake up to your consciousness.

Our beloved peer-review system was grossly abused and failed spectacularly in the case of the ELM. It is time for the machine learning experts and leaders to investigate the allegations presented here and to take corrective actions soon.

5 Easy but Proven Steps to Academic Fame

1. The Brink of Genius: Take a paper published about 20 years ago (so that the original authors have either passed away, retired, or are too well-established/generous to publicly object. Unfortunately, pioneers like Broomhead and Pao have passed away). Introduce a very minor variation, for example, by fixing one of the tunable parameters at zero (who cares if this makes the old method worse, as long as you can claim it is now different and faster). Rewrite the paper in such a way that plagiarism software cannot detect the similarity, so that you are not in any of the “IEEE 5 levels of plagiarism”. Give a completely new sensational name (hint: the word “extreme” sounds extremely sexy).
2. Publication: Submit your paper(s) to a poor quality conference or journal without citing any related previous works.
3. Salesmanship: After publishing such a paper, now it is time to sell the stolen goods! Never blush. Don't worry about ethics. Get your friends/colleagues to use your “big thing”. Put up your Matlab program for download. Organize journal special issues, conferences, etc. to promote these unethical research practices among junior researchers who would just trust your unethical publications without bothering to read the original works published in the 1980s or 1990s. Of course, the pre-requisite for a paper to be accepted in your special issues/conferences is 10s of citations for your unethically created name and publications. Invite big names to be associated with your unethically created name as advisory board members, keynote speakers, or co-authors. These people may be too busy to check the details (with a default assumption that your research is ethical) and/or too nice to say no. But, once “infected” with your unethically created name, they will be obliged to defend it for you.
4. The Smoke Screen: Should others point out the original work, you claim not to know the literature while pointing to a minor variation that you introduced in the first place. Instead of accepting that your work was almost the same as the literature and reverting back to the older works, you promote your work by: (1) repeating the tiny variation; (2) excluding the almost identical works in the list of references or citing and describing them incorrectly; (3) excluding thorough experimental comparisons with nearly identical works in the literature so that worse performance of your minute variations will not be exposed; (4) making negative statements about competing methods and positive statements about your unethically created name without solid experimental results using words like “may” or “analysis”; (5) comparing with apparently different methods. You can copy the theories and proofs derived for other methods and apply to your method (with tiny variation from those in the old literature) claim that your method has got a lot of theories while others do not have.
5. Fame: Declare yourself as a research leader so that junior researchers can follow your footsteps. Enjoy your new fortune, i.e., high citations, invited speeches, etc. You don’t need to be on the shoulders of giants, because you are a giant! All you have to do to get there is to follow these easy steps!

One can call the above steps “IP” (Intelligent Plagiarism), as opposed to stupid (verbatim) plagiarism specified by the IEEE in “5 levels”. The machine learning community should feel embarrassed if “IP” (Intelligent Plagiarism) was originally developed and/or grandiosely promoted by this community, while the community is supposed to create other (more ethical) intelligent algorithms to benefit the mankind.

In mid-July 2015, G.-B. Huang posted an email on his [email protected] emailing list. This email was forwarded to [email protected] for our responses. As usual, this email was meaningless and our remarks are available at http://elmorigin.wix.com/originofelm .

Email for feedback: [email protected]

Can it learn the NRBF also? how?

Hi, I have read the classifiers code. In the top of the code it is commented that it can learn different kernels.
How about Normalized Radial Basis Function? Any example of code?

Predicting probabilities instead of classes

Hi

I recently needed to predict the class probabilities instead of the class labels.

So I wrote a predict_proba() method, sticking to the convention used in other scikit classifiers.

I added the following which simply take considers the exponential ratios of the decision functions,
to class GenELMClassifier to the module elm.py .

• def predict_proba(self, X):

•    """Predict probability values using the model
  

•    Considers exponent of decision_function values
  

•    Parameters
  

---

•    X : {array-like, sparse matrix} of shape [n_samples, n_features]
  

•    Returns
  

---

•    C : numpy array of shape [n_samples, n_outputs]
  

•        Predicted values.
  

•    """
  

•    raw_predictions = np.exp(self.decision_function(X))
  

•    probabilities = np.zeros(raw_predictions.shape)
  

•    rows, cols = raw_predictions.shape
  

•    for row in range(0, rows):
  

•        total = sum(raw_predictions[row,:])
  

•        probabilities[row,:] = raw_predictions[row,:] / total
  

•    return probabilities
  

(The + signs are from my GIT diffs, please ignore).

I'm not overly familiar with ELMs but if you think the above is correct, feel free to add it up. Alternatively I would be happy to contribute code to the project.

GPU implementation?

Is anyone aware of any GPU implementation of ELM? This package is based on scikit-learn, so it will probably never support GPU. I am looking for a way to speed up the computation.

I use ELMclassifier on MNIST, accuracy is 0.0

from elm import ELMClassifier
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)
train = mnist.train.next_batch(100)
elmc = ELMClassifier(n_hidden=1000, activation_func='gaussian', alpha=0.0, random_state=0)
elmc.fit(train[0], train[1])
test = mnist.test.next_batch(100)
print('train acc is %g, test acc is %g ' %( elmc.score(train[0], train[1]), elmc.score(test[0], test[1])))

run and get this,
train acc is 0, test acc is 0

citation missing?

The README says that the centers and radius are taken as follows:

centers are taken uniformly from the bounding hyperrectangle of the inputs, and
radius = max(||x-c||)/sqrt(n_centers*2)

but citation [2] only talks about ELM, and [3] talks about RBF, but the centers and radius are taken in a different way. Is the solution in this implementation an idea of @dclambert or is there a citation missing?

ValueError: output_type='binary' , but y.shape = (30, 3)

Hello,

I'm using your implementation of ELMClassifier to run some experiments. I see it was implemented following sklearn coding interface, so i was trying to run your algorithm through a dataset and measure its performance with cross_val_score(), but i am getting this error:

ValueError: output_type='binary', but y.shape = (30, 3)

It has something to do with line 614, where you set:

   class_predictions = self.binarizer.inverse_transform(raw_predictions)

Do you have any clue how can i fix this problem? I am using Iris dataset (3 classes). Please let me know if i can help or even push a pull request for this.

Thanks in advance.