Getting "TypeError: Exception encountered when calling layer "tf.keras.backend.rnn" (type TFOpLambda)" when I employ the Attention Layer

[thushv89]

Getting "TypeError: Exception encountered when calling layer "tf.keras.backend.rnn" (type TFOpLambda)" when I employ the Attention Layer

[TheZaraKhan]

@thushv89 Please solve this issue

[eliashossain001]

Has anyone solved these issues?

[thushv89]

Hi,
Sorry about the delay and thanks for raising this. Yes there's an unknown issue with the tf.keras.backend.rnn operation in tensorflow>2.6. I haven't been able to look into this in too much depth. I have tried few potential fixes but they all failed. So looks like something major has shifted since tensorflow>2.6.

The easiest solution would be to downgrade to tensorflow==2.6 until the issue is understood. But will keep you posted on the findings.

[rudy-becarelli]

Hi @thushv89 any news? I just tried to use tensorflow==2.6 but i still have the same problem....

[AliMi001]

what about using the keras additive attention layer
attn_out = AdditiveAttention()([decoder_outputs, encoder_outputs])
would it be a fix ?

[thushv89]

@rudy-becarelli: Apologies about the delay. Could you try tensorflow==2.5.0?

@AliMi001: It would be a close substitute but not a complete one. For example you can see here how TensorFlow uses AdditiveAttention for implementing Bahdanau attention. However, it does not compute attention sequentially as a sequence would be processed in a RNN. Rather, it computes all outputs first and then apply attention on top of that. I'm not sure what the performance difference is, but it is different from the original Bahdanau attention proposed in the paper.

[AbhiIISc]

I have tried tensorflow==2.5.0 with no success

[thushv89]

Good news!
A fix is on the way on https://github.com/thushv89/attention_keras/tree/tf2-fix
Should work fine once merged

Let me know if there are any issues.

[ITHealer]

I can not access repo

[mogesam]

I have similar problem, is this issue solved?

[bhattaraisubash]

Try replacing
tensorflow.python.keras.layers with tensorflow.keras.layers on imports. See if it works.