EXCEPTION NetworkConstructionDependencyLoopException: Error: There is a dependency loop on layer 'accum_att_weights'.

[rwth-i6]

EXCEPTION NetworkConstructionDependencyLoopException: Error: There is a dependency loop on layer 'accum_att_weights'.

[albertz]

Unfortunately the configs are slightly buggy, and do not work correctly with a more recent RETURNN version.
It should be easy to fix, though. Change this:

"p_t_in": {"class": "eval", "from": "prev:att_weights", "eval": "tf.squeeze(tf.argmax(source(0), axis=1, output_type=tf.int32), axis=1)",
  "out_type": {"shape": (), "batch_dim_axis": 0, "dtype": "float32"}},

To:

"p_t_in": {"class": "reduce", "from": "prev:att_weights", "mode": "argmax", "axis": "t"},

Also make sure that you use the latest RETURNN version.

[manish-kumar-garg]

After making this change and using latest RETURNN version.
Getting incorrect shapes

TensorFlow exception: Incompatible shapes: [14,1,45] vs. [45,1,45]
	 [[node output/rec/att_weights/LogicalAnd_1 (defined at /home/ubuntu/rwth-i6/returnn-experiments/2018-asr-attention/librispeech/full-setup-attention/returnn/TFNetworkLayer.py:3159) ]]

stdout

[albertz]

Thank you. There was actually a real bug, which I fixed now (commit 7999f7430cb968).
The test test_rec_layer_local_att_train_and_search should cover this now.
Can you try again with latest Returnn?

[manish-kumar-garg]

It works now. Thanks!

[manish-kumar-garg]

@albertz ,
After making this change,
I can see that the model is not converging as expected.

Logs here

Loss is always around 6k-7k

[albertz]

We did three kinds of experiments in the paper:

• Training with global soft attention, and then just importing it into this local soft attention config.
• Training with global soft attention, then importing it as local soft attention, and further training a bit.
• Training with local soft attention from scratch.

@Spotlight0xff are these the configs you used for importing the model, or from scratch training? If only for importing, can you also add the configs for the from-scratch training? (Or maybe just one reasonable/representative config for that.)

I remember that the from scratch training was quite unstable and needed some more tuning.
The importing (and optionally continue training a bit) should work in any case. Did you try that?

[manish-kumar-garg]

I have not tried that yet. Will try now.

[Spotlight0xff]

Hi,

The configs in librispeech/ were all for pretrained (global attention) models (so case 1 and 2, but not 3.).
I just added a config for from-scratch training here.
The issue with your model was probably the too small learning rate which was used when we retrained the initialized models.