Training on Python data leads to nan loss

[tech-srl]

Training on Python data leads to nan loss

[urialon]

Hi @amackillop ,
Thank you for your interest in code2seq.
I'm surprised that this is happening.

Let's investigate this.
First - can you check if there are examples in the output of your extractor, in which the target prediction (the ground truth) or any of the components of the path: "source", "target" or "path" are of "zero length"?
I.e., a path of my|value,For|If|Expr, (missing target for the path) or ,For|If|Expr,my|value (missing path source), or my|value,,my|key (missing path nodes).

If there is no such example, can you check if there is a single example that causing this ("polluting" the other examples with nans), or, alternatively, any example causes this?

This can be checked by duplicating the same line that comes as an output of your python extractor by 1000 or so. So you can have a file that is made of 1000 duplications of the same line, run the following preprocessing, and train the model on this toy dataset. By doing this for 3-4 different lines, we can see if all lines cause nan, or whether just specific lines.

P.S - you can try even less than 1000 lines, I think that the only limitation is that the batch size would be larger than the size of the dataset.

Uri