chensvm/a-dual-stage-attention-based-recurrent-neural-network-for-time-series-prediction

a-dual-stage-attention-based-recurrent-neural-network-for-time-series-prediction's Issues

what's the difference between your self-defined layer "My_Dot" and "Dense" layer in Keras?

Dense layer: Wx+b,
My_Dor: Wx
?

visualize input attentions activations?

Thank you for the wonderful code. I have tried to get input attention weights using following code

    def get_activations(self, st, en, lyr_name='attn_weight_8'):

        test_x, test_y, test_label = self.fetch_data(start=st, ende=en, shuffle=False)

        s0_test = h0_test = np.zeros((test_x.shape[0], self.nn_config['m']))

        bs = test_x.shape[0]  # self.data_config['batch_size']

        intermediate_model = KModel(
            inputs=self.k_model.inputs,
            outputs=self.k_model.get_layer(name=lyr_name).output)

        h_de0_test = s_de0_test = np.zeros((test_x.shape[0], self.nn_config['p']))
        activations = intermediate_model.predict([test_x, test_y, s0_test, h0_test, s_de0_test, h_de0_test],
                                                     batch_size=bs,
                                                     verbose=1)
        return activations, test_x

I had renamed the attention weight layer as following

    def encoder_attention(self, _input, _s0, _h0):

        s = _s0
        _h = _h0
        print('encoder cell state:', s)
        # initialize empty list of outputs
        attention_weight_t = None
        for t in range(self.lookback):
            print('encoder input:', _input)
            _context = self.one_encoder_attention_step(_h, s, _input, t)  # (none,1,n)
            print('context:', _context)
            x = Lambda(lambda x: _input[:, t, :])(_input)
            x = Reshape((1, self.ins))(x)
            print('x:', x)
            _h, _, s = self.en_LSTM_cell(x, initial_state=[_h, s])
            if t != 0:
                print('attention_weight_:', t, attention_weight_t)
                attention_weight_t = Merge(mode='concat', concat_axis=1,
                                           name='attn_weight_'+str(t))([attention_weight_t,
                                                                        _context])
                # attention_weight_t = Concatenate(axis=1)([attention_weight_t, _context])
                print('salam')
            else:
                attention_weight_t = _context
            print('encoder hidden state:', _h)
            print('_:', _)
            print('encoder cell state:', s)
            print('time-step', t)
            # break

        # get the driving input series
        enc_output = Multiply(name='enc_output')([attention_weight_t, _input])  # equation 10 in paper
        print('output from encoder attention:', enc_output)
        return enc_output

and then plot the activations using following two methods

    def plot2d_act_for_a_sample(self, activations, sample=0, name=None):
        fig, axis = plt.subplots()
        fig.set_figheight(8)
        # for idx, ax in enumerate(axis):
        axis.imshow(activations[sample, :, :].transpose())
        axis.set_xlabel('lookback')
        axis.set_ylabel('inputs')
        print(self.data_config['inputs'])
        axis.set_title('Activations of all inputs at different lookbacks for sample '+ str(sample))
        if name is not None:
            plt.savefig(name + '_' + str(sample), dpi=400, bbox_inches='tight')
        plt.show()
        return

    def plot1d_act_for_a_sample(self, activations, sample=0, name=None):
        fig, axis = plt.subplots()

        for idx in range(self.lookback-1):
            axis.plot(activations[sample, idx, :].transpose(), label='lookback '+str(idx))
        axis.set_xlabel('inputs')
        axis.set_ylabel('activation weight')
        axis.set_title('Activations at different lookbacks for all inputs for sample ' + str(sample))
        if name is not None:
            plt.savefig(name + '_' + str(sample), dpi=400, bbox_inches='tight')
        plt.show()

I get 1d and 2d plots as attached with this thread. The first 81 are real inputs and second 81 are noise (randomly generated time-series). This figure is not similar to figure 3 of the paper. Can you help me make sense of these figures? Otherwise can you please show me how to get figure 3 of the original paper?

Although overall the weights for first 81 inputs are more than those of seconda 81(dummy) inputs but the dummy inputs are still contributing a lot to the model output.

some questions

I try to test this model with this sentence:
predicted_value = model.predict([input_X_test,input_Y_test,s0_test,h0_test,s_de0_test,h_de0_test])
but I cannot get good predicted value.
How does the test result perform in your code?

where is Generate_stock_data?

How to save the model?

Hi, Chen:

I download your code and run it on my computer successfully. Now , I want to save the model, I edit the code :

model.fit([input_X_train,input_Y_train,s0_train,h0_train,s_de0_train,h_de0_train],label_Y_train,epochs=epochs,batch_size=batch_size,validation_split=0.2, callbacks=[ModelCheckpoint(filepath='{val_loss:.4f}-weights.{epoch:02d}.hdf5', monitor='val_loss', save_best_only=False, mode='min',period=1)]).

But it raised an error:
TypeError: can't pickle _thread.lock objects.

Whole error information:
22656/22708 [============================>.] - ETA: 0s - loss: 21343456.8136 - mean_squared_error: 21343456.8136C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_1 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'input_3:0' shape=(?, 20) dtype=float32>, <tf.Tensor 'input_2:0' shape=(?, 20) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_1 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1/TensorArrayReadV3:0' shape=(?, 20) dtype=float32>, <tf.Tensor 'lstm_1/while/Exit_3:0' shape=(?, 20) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_1 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1_1/TensorArrayReadV3:0' shape=(?, 20) dtype=float32>, <tf.Tensor 'lstm_1_1/while/Exit_3:0' shape=(?, 20) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_1 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1_2/TensorArrayReadV3:0' shape=(?, 20) dtype=float32>, <tf.Tensor 'lstm_1_2/while/Exit_3:0' shape=(?, 20) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_1 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1_3/TensorArrayReadV3:0' shape=(?, 20) dtype=float32>, <tf.Tensor 'lstm_1_3/while/Exit_3:0' shape=(?, 20) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_1 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1_4/TensorArrayReadV3:0' shape=(?, 20) dtype=float32>, <tf.Tensor 'lstm_1_4/while/Exit_3:0' shape=(?, 20) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_1 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1_5/TensorArrayReadV3:0' shape=(?, 20) dtype=float32>, <tf.Tensor 'lstm_1_5/while/Exit_3:0' shape=(?, 20) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_1 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1_6/TensorArrayReadV3:0' shape=(?, 20) dtype=float32>, <tf.Tensor 'lstm_1_6/while/Exit_3:0' shape=(?, 20) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_1 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_1_7/TensorArrayReadV3:0' shape=(?, 20) dtype=float32>, <tf.Tensor 'lstm_1_7/while/Exit_3:0' shape=(?, 20) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'input_4:0' shape=(?, 30) dtype=float32>, <tf.Tensor 'input_5:0' shape=(?, 30) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_2/TensorArrayReadV3:0' shape=(?, 30) dtype=float32>, <tf.Tensor 'lstm_2/while/Exit_3:0' shape=(?, 30) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_2_1/TensorArrayReadV3:0' shape=(?, 30) dtype=float32>, <tf.Tensor 'lstm_2_1/while/Exit_3:0' shape=(?, 30) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_2_2/TensorArrayReadV3:0' shape=(?, 30) dtype=float32>, <tf.Tensor 'lstm_2_2/while/Exit_3:0' shape=(?, 30) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_2_3/TensorArrayReadV3:0' shape=(?, 30) dtype=float32>, <tf.Tensor 'lstm_2_3/while/Exit_3:0' shape=(?, 30) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_2_4/TensorArrayReadV3:0' shape=(?, 30) dtype=float32>, <tf.Tensor 'lstm_2_4/while/Exit_3:0' shape=(?, 30) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_2_5/TensorArrayReadV3:0' shape=(?, 30) dtype=float32>, <tf.Tensor 'lstm_2_5/while/Exit_3:0' shape=(?, 30) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_2_6/TensorArrayReadV3:0' shape=(?, 30) dtype=float32>, <tf.Tensor 'lstm_2_6/while/Exit_3:0' shape=(?, 30) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py:2362: UserWarning: Layer lstm_2 was passed non-serializable keyword arguments: {'initial_state': [<tf.Tensor 'lstm_2_7/TensorArrayReadV3:0' shape=(?, 30) dtype=float32>, <tf.Tensor 'lstm_2_7/while/Exit_3:0' shape=(?, 30) dtype=float32>]}. They will not be included in the serialized model (and thus will be missing at deserialization time).
str(node.arguments) + '. They will not be included '
Traceback (most recent call last):
File "D:\sunfan\work\python\code\DARNN\dual_attention.py", line 214, in
model.fit([input_X_train,input_Y_train,s0_train,h0_train,s_de0_train,h_de0_train],label_Y_train,epochs=epochs,batch_size=batch_size,validation_split=0.2, callbacks=[ModelCheckpoint(filepath='{val_loss:.4f}-weights.{epoch:02d}.hdf5', monitor='val_loss', save_best_only=False, mode='min',period=1)])
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py", line 1669, in fit
validation_steps=validation_steps)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\training.py", line 1226, in _fit_loop
callbacks.on_epoch_end(epoch, epoch_logs)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\callbacks.py", line 76, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\callbacks.py", line 429, in on_epoch_end
self.model.save(filepath, overwrite=True)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py", line 2574, in save
save_model(self, filepath, overwrite, include_optimizer)
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\models.py", line 111, in save_model
'config': model.get_config()
File "C:\ProgramData\Anaconda3\lib\site-packages\keras\engine\topology.py", line 2415, in get_config
return copy.deepcopy(config)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 215, in _deepcopy_list
append(deepcopy(a, memo))
File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "C:\ProgramData\Anaconda3\lib\copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "C:\ProgramData\Anaconda3\lib\copy.py", line 220, in
y = [deepcopy(a, memo) for a in x]
File "C:\ProgramData\Anaconda3\lib\copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 180, in deepcopy
y = _reconstruct(x, memo, *rv)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 280, in _reconstruct
state = deepcopy(state, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 150, in deepcopy
y = copier(x, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "C:\ProgramData\Anaconda3\lib\copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle _thread.lock objects

How to fix the problem or another way to save the model?Thank u!

AttributeError: 'tuple' object has no attribute 'layer'

---

AttributeError Traceback (most recent call last)
in
5 s_de0 = Input(shape=(n_sde0,))
6 Y = Input(shape=(T-1,1))
----> 7 X_ = encoder_attention(T,X,s0,h0)
8 print('X_:',X_)
9 X_ = Reshape((T,n))(X_)

in encoder_attention(T, X, s0, h0)
29 for t in range(T):
30 print('X:', X)
---> 31 context = one_encoder_attention_step(h,s,X) #(none,1,n)
32 print('context:',context)
33 x = Lambda(lambda x: X[:,t,:])(X)

in one_encoder_attention_step(h_prev, s_prev, X)
10 result1 = en_densor_We(concat) #(none,1,T)
11 result1 = RepeatVector(X.shape[2],)(result1) #(none,n,T)
---> 12 X_temp = My_Transpose(axis=(0,2,1))(X) #X_temp(None,n,T)
13 result2 = My_Dot(T)(X_temp) # (none,n,T) Ue(T,T)
14 result3 = Add()([result1,result2]) #(none,n,T)

~\Anaconda3\envs\chensvm\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py in call(self, inputs, *args, **kwargs)
879 kwargs.pop('mask')
880 inputs, outputs = self.set_connectivity_metadata(
--> 881 inputs, outputs, args, kwargs)
882 self._handle_activity_regularization(inputs, outputs)
883 self._set_mask_metadata(inputs, outputs, input_masks)

~\Anaconda3\envs\chensvm\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py in set_connectivity_metadata(self, inputs, outputs, args, kwargs)
2041 # This updates the layer history of the output tensor(s).
2042 self._add_inbound_node(
-> 2043 input_tensors=inputs, output_tensors=outputs, arguments=arguments)
2044 return inputs, outputs
2045

~\Anaconda3\envs\chensvm\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py in _add_inbound_node(self, input_tensors, output_tensors, arguments)
2057 """
2058 inbound_layers = nest.map_structure(lambda t: t._keras_history.layer,
-> 2059 input_tensors)
2060 node_indices = nest.map_structure(lambda t: t._keras_history.node_index,
2061 input_tensors)

~\Anaconda3\envs\chensvm\lib\site-packages\tensorflow_core\python\util\nest.py in map_structure(func, *structure, **kwargs)
534
535 return pack_sequence_as(
--> 536 structure[0], [func(*x) for x in entries],
537 expand_composites=expand_composites)
538

~\Anaconda3\envs\chensvm\lib\site-packages\tensorflow_core\python\util\nest.py in (.0)
534
535 return pack_sequence_as(
--> 536 structure[0], [func(*x) for x in entries],
537 expand_composites=expand_composites)
538

~\Anaconda3\envs\chensvm\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py in (t)
2056 call method of the layer at the call that created the node.
2057 """
-> 2058 inbound_layers = nest.map_structure(lambda t: t._keras_history.layer,
2059 input_tensors)
2060 node_indices = nest.map_structure(lambda t: t._keras_history.node_index,

AttributeError: 'tuple' object has no attribute 'layer'

any idea how to solve this ?

ValueError: Operands could not be broadcast together with shapes (1, Dimension(810)) (10, 81)

importing from DARNN import dual_attention

gives the error

ValueError: Operands could not be broadcast together with shapes (1, Dimension(810)) (10, 81)

---

ValueError Traceback (most recent call last)
in
----> 1 from DARNN import dual_attention

~/repos/A-Dual-Stage-Attention-Based-Recurrent-Neural-Network-for-Time-Series-Prediction/DARNN/dual_attention.py in
193 s_de0 = Input(shape=(n_sde0,))
194 Y = Input(shape=(T-1,1))
--> 195 X_ = encoder_attention(T,X,s0,h0)
196 print('X_:',X_)
197 X_ = Reshape((T,n))(X_)

~/repos/A-Dual-Stage-Attention-Based-Recurrent-Neural-Network-for-Time-Series-Prediction/DARNN/dual_attention.py in encoder_attention(T, X, s0, h0)
141 # break
142
--> 143 X_ = Multiply()([attention_weight_t,X])
144 print('return X:',X_)
145 return X_

~/anaconda3/envs/Dual-Stage-Attention-for-TS/lib/python3.6/site-packages/keras/engine/topology.py in call(self, inputs, **kwargs)
592 self.build(input_shapes[0])
593 else:
--> 594 self.build(input_shapes)
595 self.built = True
596

~/anaconda3/envs/Dual-Stage-Attention-for-TS/lib/python3.6/site-packages/keras/layers/merge.py in build(self, input_shape)
88 else:
89 shape = input_shape[i][1:]
---> 90 output_shape = self._compute_elemwise_op_output_shape(output_shape, shape)
91 # If the inputs have different ranks, we have to reshape them
92 # to make them broadcastable.

~/anaconda3/envs/Dual-Stage-Attention-for-TS/lib/python3.6/site-packages/keras/layers/merge.py in _compute_elemwise_op_output_shape(self, shape1, shape2)
59 raise ValueError('Operands could not be broadcast '
60 'together with shapes ' +
---> 61 str(shape1) + ' ' + str(shape2))
62 output_shape.append(i)
63 return tuple(output_shape)

ValueError: Operands could not be broadcast together with shapes (1, Dimension(810)) (10, 81)

what the difference My_Transpose with Permute

in my view, your code My_Transpose seem have the same function with the Keras.layer.Permute

merge

What version of tensorflow and keras were used with this? Merge can no longer be imported and concatenate needs to be used but isn't working