thushv89/autoencorder_simple

Hi there! Amazing fellow human being 👋

I am a data scientist and I love getting my hands 🤚 and feet 🦶 wet in problem infested ☠ murky waters 🚤 of data, to unearth invaluable information and actionable insights 💎.

What do I love?

• 🐍 Python
• 🤖 Machine Learning, Deep Learning, Artificial Intelligence
• 🧮 TensorFlow

Some of my publications

• 📙 Natural Language Processing with TensorFlow 2 (Packt)
• 📙 TensorFlow 2 in Action (Manning)
• 🎬 Machine Translation in Python (DataCamp)essing-TensorFlow-language-ebook/dp/B077Q3VZFR)
• 📑 Medium Articles
• 🎬 DeepLearningHero Youtube Channel

autoencorder_simple's Issues

Autoencoder.py gives very similar looking reconstructions for all the inputs

This might have something to do with cost function. as intuitively I feel like the cost function is trying to come up with an average output which suits all the inputs therefore end up looking very similar

There is lot of noise in the visualized features

Errors while running optimize.minimize on laptop

Err1: Bad direction in the line search; refresh the lbfgs memory and restart the iteration.

Err2: Line search cannot locate an adequate point after 20 function and gradient evaluations. Previous x, f and g restored.
Possible causes: 1 error in function or gradient evaluation; 2 rounding error dominate computation.

visualization of hidden layers with max activation (nlopt) - return Nan for dot(x,W)

Explained in this.
http://mail.scipy.org/pipermail/numpy-discussion/2013-August/067506.html

large floating point multiplication causes this (float32/64)

Fix the issue with softmax classifier

Fixed the issue with the softmax classifier. The problem was, in the gradient descent I was doing this
delta[i] = y_vec[i]-actv[k] where y=k (y_vec is vectorized form of y, actv is current output activation vector for y)

This is WRONG! Correct implementation is
delta[i] = y_vec[i] - actv[i]

Finish implementing image plotting script

Implement Feature Visualizing feature GPU code

Implement gradient checking to make sure GD is correct (autoencoder_with_l_bfgs)

Reconstruction of learnt images was always plain gray

It seems like an issue with the used weight update rule (http://ufldl.stanford.edu/wiki/index.php/Backpropagation_Algorithm)
Because once I changed the weight update rule to (http://www.cse.unsw.edu.au/~cs9417ml/MLP2/) it started working much better

how to fix - compile dir mod.cpp error LONG_BIT definition

libpython was missing (A dependency of theano)

https://www.kaggle.com/c/otto-group-product-classification-challenge/forums/t/13973/a-few-tips-to-install-theano-on-windows-64-bits/80209

Disconnected gradient Issue THeano

theano.gradient.DisconnectedInputError: grad method was asked to compute the gradient with respect to a variable that is not part of the computational graph of the cost, or is used only by a non-differentiable operator: W1

UnusedInputError(msg % (inputs.index(i), i.variable, err_msg)) when using index in theano.function for Autoencoder

UnusedInputError(msg % (inputs.index(i), i.variable, err_msg))
theano.compile.function_module.UnusedInputError: theano.function was asked to create a function computing outputs given certain inputs, but the provided input variable at index 0 is not part of the computational graph needed to compute the outputs: index.

Implement Denoising Autoencoders

Accuracy with different hidden layer sizes

Hidden layers [400,225,100] pre epochs = 25 fine_tuning = 50 batch_size=10
Test Error: 0.0171

Hidden layers [400]
Test Error: 0.018

Implementing dropout technique

http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf

Compare Cost Functions: Sqr Error and Negative log likelyhood

The cost function can be changed by changing the argument "cost_fn" to cost functions in SparseAutoencoder and SoftmaxClassifier

Read about pre-constraining in optimize.minimize (scipy)

Implement feature visualization for higher layers

related to issue #20

Learnt Features look very random in layer 2 and 3

Weight decay term reduces the accuracy significantly

Implement Stochastic GD for autoencoder_l_bfgs.py

can't multiply sequence by non-int of type 'float'

This issue was encountered in
updates = [
(param, param - l_rate*gparams)
for param, gparam in zip(self.theta,gparams)]

which should be
updates = [
(param, param - l_rate*gparam)
for param, gparam in zip(self.theta,gparams)]

If this error is occuring it's probably because you're trying to multiply a list by a float where you should multiply a scalar by a float

Start implementing stacked autoencoder

Introduce better regularization

Introduce weight-decay, sparcity

Test gradient check values given by scipy check_grad and manual grad check

Implement Autoencoder with Theano

All the test cases get the same value - Iris dataset

The problem has been in the following code

for i in range(0, iter):
for idx in range(0, X.size[1]):
....
p_deriv_W2 = np.dot(delta3[:, None], np.transpose(a2[:, None]))
p_deriv_b2 = delta3

       delta_W2 = delta_W2 + p_deriv_W2
       delta_b2 = delta_b2 + p_deriv_b2

       self.W2 = self.W2 - (alpha * delta_W2) + (M * prev_delta_W2)
       self.b2 = self.b2 - (alpha * delta_b2) + (M * prev_delta_b2)

Having code like this inside training case loop causes delta_W2 to grow in very abnormal ways. The solutions is to have the following

       delta_W2 = p_deriv_W2
       delta_b2 = p_deriv_b2