fchollet/keras-resources

Directory of tutorials and open-source code repositories for working with Keras, the Python deep learning library

keras-resources's Issues

Research

Working

Link broken

Stateful LSTM, Siamese Network link is broken.

Adding tutorials?

Custom layer tutorial: https://keunwoochoi.wordpress.com/2016/11/18/for-beginners-writing-a-custom-keras-layer/
Callback tutorial: https://keunwoochoi.wordpress.com/2016/07/16/keras-callbacks/

These are very simple ones, please add them if you think they're suitable.

How to hack keras!

MultiHeadAttention

I would like to match the results of the self_attention() function on page 339 of the Keras book, Deep learning with Python, second edition, with the those of the MultiHeadAttention() example just below. I wrote an example with the same input and I have different results. Can somebody explain why?

import numpy as np
from scipy.special import softmax
from tensorflow.keras.layers import MultiHeadAttention


def self_attention(input_sequence):
    output = np.zeros(shape=input_sequence.shape)
    # The output will consist of contextual embeddinsgs of the same shape
    for i, pivot_vector in enumerate(input_sequence):
        scores = np.zeros(shape=(len(input_sequence),))
        for j, vector in enumerate(input_sequence):
            scores[j] = np.dot(pivot_vector, vector.T)  # Q K^T
        scores /= np.sqrt(input_sequence.shape[1])  # sqrt(d_k)
        scores = softmax(scores)  # softmax(Q K^T / sqrt(d_k))
        print(i, scores)
        new_pivot_representation = np.zeros(shape=pivot_vector.shape)
        for j, vector in enumerate(input_sequence):
            new_pivot_representation += vector * scores[j]
        output[i] = new_pivot_representation
    return output


test_input_sequence = np.array([[[1.0, 0.0, 0.0, 1.0],
                                 [0.0, 1.0, 0.0, 0.0],
                                 [0.0, 1.0, 1.0, 1.0]]])

test_input_sequence.shape
# (1, 3, 4)

self_attention(test_input_sequence[0])
"""
returns
[[0.50648039 0.49351961 0.30719589 0.81367628]
 [0.23269654 0.76730346 0.38365173 0.61634827]
 [0.21194156 0.78805844 0.57611688 0.78805844]]
 
the attention scores being:
[0.50648039 0.18632372 0.30719589]
[0.23269654 0.38365173 0.38365173]
[0.21194156 0.21194156 0.57611688]
"""
att_layer = MultiHeadAttention(num_heads=1,
                               key_dim=4,
                               use_bias=False,
                               attention_axes=(1,))

att_layer(test_input_sequence,
          test_input_sequence,
          test_input_sequence,
          return_attention_scores=True)

"""
returns 
array([[[-0.46123487,  0.36683324, -0.47130704, -0.00722525],
        [-0.49571565,  0.37488416, -0.52883905, -0.02713571],
        [-0.4566634 ,  0.38055322, -0.45884743, -0.00156384]]],
      dtype=float32)
      
and the attention scores
array([[[[0.31446996, 0.36904442, 0.3164856 ],
         [0.34567958, 0.2852166 , 0.36910382],
         [0.2934979 , 0.3996053 , 0.30689687]]]], dtype=float32)>)
"""

Keras + Tensorflow + InceptionV3 example does not work

Hi,

I've seen that the inception example doesn't exists. The following URL doesn't exists: https://github.com/fchollet/keras/blob/master/examples/inception_v3.py

Is that example the same that you presented at TensorFlow Dev Summit 2017 (https://www.youtube.com/watch?v=UeheTiBJ0Io)?

Thank you in advance.

Regards,
Rubén