dandxy89/deeplearningreading

Deep Learning and Machine Learning mini-projects. Current Project: Deepmind Attentive Reader (rc-data)

Hello! I'm Dan. 👋

• 🔭 I’m currently working as a Rust Engineer / Quant Developer at Laser Digital
• 🌱 I’m currently learning about Market Making, Rust and SIMD.
• 🤔 I’m looking for help with Market Making and Linear Programming / Optimisation.
• 📫 How to reach me: LinkedIn or Emails

deeplearningreading's Issues

Adding more example of NLP classifications

Drop me a message (email) or comment here, if you require any additional examples.

Why word2vec is performed on train and test?

Dear

I think the word2vec should be applied only on train data not the combined (train + test) to simulate real life situations where test data is not available.

imdb_embedding_w2v.py
line 160: model.build_vocab(combined)

If so, how to initialize "embedding_weights"? what if the embedding_weights come from totally different text - different vocabulary - such as word2vec model by google or any other independent offline trained model?

imdb_embedding_w2v.py
lione 195: weights=[embedding_weights]

Regards

lets check your code for over fitting per this paper

new drop out for embeddings layer : conventional drop outs leads to overfititng
http://arxiv.org/abs/1512.05287
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Yarin Ga

TypeError: 'numpy.float64' object is not iterable

(tensorflow) admin@macbookproloreto:~/Developmemt/ParisiLabs/ML/DeepLearning_MachineLearning/EmbeddingKeras$ python imdb_embedding_w2v.py 
Using TensorFlow backend.
Loading Data...
Tokenising...
Training a Word2vec model...
Transform the Data...
Setting up Arrays for Keras Embedding Layer...
Creating Datesets...
Pad sequences (samples x time)
('X_train shape:', (25000, 100))
('X_test shape:', (25000, 100))
Convert labels to Numpy Sets...
Defining a Simple Keras Model...
Compiling the Model...
/Users/admin/tensorflow/lib/python2.7/site-packages/keras/models.py:540: UserWarning: "class_mode" argument is deprecated, please remove it.
  warnings.warn('"class_mode" argument is deprecated, '
Train...
/Users/admin/tensorflow/lib/python2.7/site-packages/keras/models.py:603: UserWarning: The "show_accuracy" argument is deprecated, instead you should pass the "accuracy" metric to the model at compile time:
`model.compile(optimizer, loss, metrics=["accuracy"])`
  warnings.warn('The "show_accuracy" argument is deprecated, '
Train on 25000 samples, validate on 25000 samples
Epoch 1/2
25000/25000 [==============================] - 1219s - loss: 0.4462 - val_loss: 0.4233
Epoch 2/2
25000/25000 [==============================] - 1067s - loss: 0.2813 - val_loss: 0.3266
Evaluate...
/Users/admin/tensorflow/lib/python2.7/site-packages/keras/models.py:644: UserWarning: The "show_accuracy" argument is deprecated, instead you should pass the "accuracy" metric to the model at compile time:
`model.compile(optimizer, loss, metrics=["accuracy"])`
  warnings.warn('The "show_accuracy" argument is deprecated, '
25000/25000 [==============================] - 210s     
Traceback (most recent call last):
  File "imdb_embedding_w2v.py", line 213, in <module>
    show_accuracy=True)
TypeError: 'numpy.float64' object is not iterable

using

$ pip list | grep gensim
gensim (0.12.4)
$ pip list | grep numpy
numpy (1.11.1)

Verification

Verify that the model located here has been correctly setup with Attention?

Replicating the work by Deepmind - Attentive Reader

Language:

Python (Keras w/ Theano backend)

Model

The Attentive Reader as described in the paper utilizes an attention mechanism inspired by recent results in translation and image recognition.

The attention mechanism that we have employed is just one instantiation of a very general idea
which can be further exploited. However, the incorporation of world knowledge and multi-document
queries will also require the development of attention and embedding mechanisms whose complex-
ity to query does not scale linearly with the data set size. There are still many queries requiring
complex inference and long range reference resolution that our models are not yet able to answer.
As such our data provides a scalable challenge that should support NLP research into the future. Further, significantly bigger training data sets can be acquired using the techniques we have described, undoubtedly allowing us to train more expressive and accurate models.

Info:

Currently training the model via CPU - I intend to move the processing over to a GPU once I have can see that performance is sufficiently worthwhile.

Performance (Accuracy %):

Epoch	Validation	Test
1	0.251	0.219
2	0.807	0.768
3	1.363	1.316
4	1.919	1.864
5	2.475	2.412
6	3.603	3.556
7	4.731	4.699

Once I have achieved > 10% I intend to move the model over too an AWS instance.

Keras implementation of Deepmind's Attentive Reader

Language:

Python (Keras w/ Theano backend)

Model

The Attentive Reader as described in the paper utilizes an attention mechanism inspired by recent results in translation and image recognition.

The attention mechanism that we have employed is just one instantiation of a very general idea
which can be further exploited. However, the incorporation of world knowledge and multi-document
queries will also require the development of attention and embedding mechanisms whose complex-
ity to query does not scale linearly with the data set size. There are still many queries requiring
complex inference and long range reference resolution that our models are not yet able to answer.
As such our data provides a scalable challenge that should support NLP research into the future. Further, significantly bigger training data sets can be acquired using the techniques we have described, undoubtedly allowing us to train more expressive and accurate models.

Info:

Currently training the model via CPU - I intend to move the processing over to a GPU once I have can see that performance is sufficiently worthwhile.

Performance (Accuracy %):

Epoch	Validation	Test
1	0.251	0.219
2	0.807	0.768
3	1.363	1.316
4	1.919	1.864
5	2.475	2.412
6	3.603	3.556
7	4.731	4.699

Once I have achieved > 10% I intend to move the model over too an AWS instance.

Tied parameters for question and answer in attention model

Thank you for a such a nice implementation. In this implementation you have different weights for both question and answer BiLSTMs. I was wondering how to implement similar attention model with tied BiLSTM parameters in keras. Any pointer will be helpful. Thanks !!!

Getting error AttributeError: 'module' object has no attribute 'control_flow_ops'

python imdb_embedding_w2v.py
Using TensorFlow backend.
Loading Data...
Tokenising...
Training a Word2vec model...
Transform the Data...
Setting up Arrays for Keras Embedding Layer...
Creating Datesets...
Pad sequences (samples x time)
('X_train shape:', (25000, 100))
('X_test shape:', (25000, 100))
Convert labels to Numpy Sets...
Defining a Simple Sequential Keras Model...
Traceback (most recent call last):
File "imdb_embedding_w2v.py", line 205, in
model.add(Dropout(0.3))
File "/home/shakti/.virtualenvs/keras/local/lib/python2.7/site-packages/keras/models.py", line 308, in add
output_tensor = layer(self.outputs[0])
File "/home/shakti/.virtualenvs/keras/local/lib/python2.7/site-packages/keras/engine/topology.py", line 514, in call
self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
File "/home/shakti/.virtualenvs/keras/local/lib/python2.7/site-packages/keras/engine/topology.py", line 572, in add_inbound_node
Node.create_node(self, inbound_layers, node_indices, tensor_indices)
File "/home/shakti/.virtualenvs/keras/local/lib/python2.7/site-packages/keras/engine/topology.py", line 149, in create_node
output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
File "/home/shakti/.virtualenvs/keras/local/lib/python2.7/site-packages/keras/layers/core.py", line 90, in call
x = K.in_train_phase(K.dropout(x, self.p, noise_shape), x)
File "/home/shakti/.virtualenvs/keras/local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 1304, in in_train_phase
x = tf.python.control_flow_ops.cond(tf.cast(_LEARNING_PHASE, 'bool'),
AttributeError: 'module' object has no attribute 'control_flow_ops'

Using following version of Keras

pip show keras

Metadata-Version: 2.0
Name: Keras
Version: 1.1.0