johnsmithm/multi-heads-attention-image-classification

Multi heads attention for image classification

Attention is all you need: A Keras Implementation

Using attention to increase image classification accuracy. Inspired from "Attention is All You Need" (Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, arxiv, 2017).

The medium article can be found here.

multi-heads-attention-image-classification's Issues

accuracy not improving

Hey I tried implementing your model.... but I could get an accuracy of just 0.0987.
What is the error?

Batch Dot and Reshape error

There's a batch dot error that shows at Line 34, axes given 4,3 that's out of range error. It should be 3,2 with the given arrays as [64, 8, 8] and [64, 8, 64].

After resolving tabove error, I am getting a Reshape error. Model summary prints fine. The error is :

Traceback (most recent call last):
  File "multi-heads-attention-mnist.py", line 121, in <module>
    validation_data=(X_test, Y_test)
  File "/home/intern_akash/akash_hpc/lib/python3.6/site-packages/keras/engine/training.py", line 1239, in fit
    validation_freq=validation_freq)
  File "/home/intern_akash/akash_hpc/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 196, in fit_loop
    outs = fit_function(ins_batch)
  File "/home/intern_akash/akash_hpc/lib/python3.6/site-packages/tensorflow/python/keras/backend.py", line 3292, in __call__
    run_metadata=self.run_metadata)
  File "/home/intern_akash/akash_hpc/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1458, in __call__
    run_metadata_ptr)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: Input to reshape is a tensor with 1146617856 values, but the requested shape has 884736
	 [[{{node model_1/reshape_5/Reshape}}]]
	 [[metrics/accuracy/Identity/_243]]
  (1) Invalid argument: Input to reshape is a tensor with 1146617856 values, but the requested shape has 884736
	 [[{{node model_1/reshape_5/Reshape}}]]

Can't pickle module objects

I can't save the keras model by calling model.save(model_path) where model is a keras model and model_path is the directory.

If I remove the attention block, then everything will be back to normal.
Though tensorflow saver does work, is there any way I can have a .h5 keras model?

Dimensionality error inside MultiHeadsAttModel

Inside the definition of MultiHeadsAttModel function, the dimension of att variable turns out to be (l, dv, dv) and not (l, nv, nv) as mentioned with the output_shape parameter.

multi-heads-attention-image-classification/multi-heads-attention-mnist.py

Lines 29 to 31 in 16a0231

	att = Lambda(lambda x: K.batch_dot(x[0],x[1] ,axes=[-1,-1]) / np.sqrt(dv),
	output_shape=(l, nv, nv))([q,k])# l, nv, nv
	att = Lambda(lambda x: K.softmax(x) , output_shape=(l, nv, nv))(att)

output_shape itself is not relevant with tensorflow backend hence it does not raise an error, and I am able to reproduce the results. However, I am not sure if the shape of att variable was intended to be (l, nv, nv)?