Questions about tf.errors.OutOfRangeError

[lishi0927]

Questions about tf.errors.OutOfRangeError

[urialon]

Hi @lishi0927 ,
Yes, this exception is thrown automatically by TensorFlow after reading the entire file, in both training and testing.
During training, if the training set is small and you want to read it multiple times before stopping for validation, you can increase this value:
https://github.com/tech-srl/code2seq/blob/master/config.py#L6

Which will make the exception be throw after training multiple times of reading the file.

Best,
Uri

[lishi0927]

Thank you for your reply.
I have tested in my small datasets, which have only 40 tuples. When I train the network, after the first 100 epoches, the evaluation always return the error is 0.0000 and the time is 0h0m0s. I want to know the loss is really 0.0 for all the 40 tuples or just one tuple then they will skip out the loop;
I set config.SAVE_EVERY_EPOCHS = 200;

[urialon]

You can also verify whether evaluation actually happened by inspecting the file log.txt which is saved in the model directory. If the file is empty, there is some issue with the data file and no example was tested.

[lishi0927]

Thank you, I see the log file;
And another question, I change a few of your codes and want to try the code vector for another research, but I find the loss function doesn't decline;
I want to ask how many epoches does the loss function decline in your applications?
And could you give me some advice to solve this problem for learning?

[urialon]

Does it not decrease at all?
You mean the training loss?

[lishi0927]

when I set USE_MOMENTUM = True, the loss will decrease a little;

For example, " the accuracy after * epoches: " is the sum of loss in the evaluation dataset, and " the average loss " in the following lines is the training loss;

2019-12-05 15:11:49.130519: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-12-05 15:11:49.495705: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Finished 1 epochs
Evaluation time: 0h0m21s
Accuracy after 1 epochs: 14.58137
Saved after 1 epochs in: model/test_dataset-model/model_iter1
Average loss at batch 250: 0.136803, throughput: 2 samples/sec
Finished 1 epochs
Evaluation time: 0h0m17s
Accuracy after 2 epochs: 14.57373
Saved after 2 epochs in: model/test_dataset-model/model_iter2
Average loss at batch 500: 0.130142, throughput: 2 samples/sec
Finished 1 epochs
Evaluation time: 0h0m17s
Accuracy after 3 epochs: 14.57148
Saved after 3 epochs in: model/test_dataset-model/model_iter3
Average loss at batch 750: 0.137707, throughput: 2 samples/sec
Finished 1 epochs
Evaluation time: 0h0m17s
Accuracy after 4 epochs: 14.57025
Saved after 4 epochs in: model/test_dataset-model/model_iter4
Average loss at batch 1000: 0.129377, throughput: 2 samples/sec
Finished 1 epochs
Evaluation time: 0h0m17s
Accuracy after 5 epochs: 14.56951
Saved after 5 epochs in: model/test_dataset-model/model_iter5
Average loss at batch 1250: 0.137484, throughput: 2 samples/sec
Finished 1 epochs
Evaluation time: 0h0m17s
Accuracy after 6 epochs: 14.56908
Saved after 6 epochs in: model/test_dataset-model/model_iter6
Average loss at batch 1500: 0.136327, throughput: 2 samples/sec
Finished 1 epochs
Evaluation time: 0h0m17s
Accuracy after 7 epochs: 14.56876
Saved after 7 epochs in: model/test_dataset-model/model_iter7
Average loss at batch 1750: 0.143147, throughput: 2 samples/sec
Finished 1 epochs
Evaluation time: 0h0m17s
Accuracy after 8 epochs: 14.56858

[urialon]

Hi,
Since you're using a dataset of only 40 examples, it doesn't make much sense to evaluate after every epoch.
You can try to increase the value of config.SAVE_EVERY_EPOCHS to 100 or 500 (which will also evaluate the model once every X epochs).

[lishi0927]

Thank you for your patient reply.
I have tried the SAVE_EVERY_EPOCHES = 100, and the training ends after 400 epoches due to "not improved for 200 epochs, stopping training", I find the loss doesn't change in the validation dataset.

Initalized variables
Started reader...
2019-12-05 20:59:29.204152: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2019-12-05 20:59:29.562570: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
Average loss at batch 250: 0.136985, throughput: 2 samples/sec
Average loss at batch 500: 0.134149, throughput: 3 samples/sec
Average loss at batch 750: 0.136165, throughput: 3 samples/sec
Average loss at batch 1000: 0.129749, throughput: 3 samples/sec
Average loss at batch 1250: 0.137963, throughput: 3 samples/sec
Average loss at batch 1500: 0.140283, throughput: 3 samples/sec
Average loss at batch 1750: 0.142418, throughput: 3 samples/sec
Average loss at batch 2000: 0.139750, throughput: 3 samples/sec
Average loss at batch 2250: 0.132917, throughput: 3 samples/sec
Average loss at batch 2500: 0.138506, throughput: 3 samples/sec
Average loss at batch 2750: 0.130794, throughput: 3 samples/sec
Average loss at batch 3000: 0.136104, throughput: 3 samples/sec
Average loss at batch 3250: 0.137308, throughput: 3 samples/sec
Average loss at batch 3500: 0.140197, throughput: 3 samples/sec
Average loss at batch 3750: 0.149420, throughput: 3 samples/sec
Average loss at batch 4000: 0.133150, throughput: 3 samples/sec
Average loss at batch 4250: 0.132581, throughput: 3 samples/sec
Average loss at batch 4500: 0.136885, throughput: 3 samples/sec
Average loss at batch 4750: 0.132741, throughput: 3 samples/sec
Average loss at batch 5000: 0.136213, throughput: 3 samples/sec
Average loss at batch 5250: 0.142391, throughput: 3 samples/sec
Average loss at batch 5500: 0.139430, throughput: 3 samples/sec
Average loss at batch 5750: 0.141992, throughput: 3 samples/sec
Average loss at batch 6000: 0.131444, throughput: 3 samples/sec
Average loss at batch 6250: 0.138770, throughput: 3 samples/sec
Average loss at batch 6500: 0.130596, throughput: 3 samples/sec
Average loss at batch 6750: 0.137449, throughput: 3 samples/sec
Average loss at batch 7000: 0.139042, throughput: 3 samples/sec
Average loss at batch 7250: 0.141805, throughput: 3 samples/sec
Average loss at batch 7500: 0.146046, throughput: 3 samples/sec
Average loss at batch 7750: 0.127546, throughput: 3 samples/sec
Average loss at batch 8000: 0.138172, throughput: 3 samples/sec
Average loss at batch 8250: 0.130903, throughput: 3 samples/sec
Average loss at batch 8500: 0.137237, throughput: 3 samples/sec
Average loss at batch 8750: 0.140205, throughput: 3 samples/sec
Average loss at batch 9000: 0.138787, throughput: 3 samples/sec
Average loss at batch 9250: 0.146157, throughput: 3 samples/sec
Average loss at batch 9500: 0.133913, throughput: 3 samples/sec
Average loss at batch 9750: 0.132571, throughput: 3 samples/sec
Average loss at batch 10000: 0.136381, throughput: 3 samples/sec
Average loss at batch 10250: 0.131310, throughput: 3 samples/sec
Average loss at batch 10500: 0.138996, throughput: 3 samples/sec
Average loss at batch 10750: 0.142923, throughput: 3 samples/sec
Average loss at batch 11000: 0.139602, throughput: 3 samples/sec
Average loss at batch 11250: 0.142040, throughput: 3 samples/sec
Average loss at batch 11500: 0.131191, throughput: 3 samples/sec
Average loss at batch 11750: 0.138803, throughput: 3 samples/sec
Average loss at batch 12000: 0.129391, throughput: 3 samples/sec
Average loss at batch 12250: 0.136524, throughput: 3 samples/sec
Average loss at batch 12500: 0.142365, throughput: 3 samples/sec
Average loss at batch 12750: 0.142322, throughput: 3 samples/sec
Average loss at batch 13000: 0.139883, throughput: 3 samples/sec
Average loss at batch 13250: 0.131158, throughput: 3 samples/sec
Average loss at batch 13500: 0.138137, throughput: 3 samples/sec
Average loss at batch 13750: 0.130916, throughput: 3 samples/sec
Average loss at batch 14000: 0.137943, throughput: 3 samples/sec
Average loss at batch 14250: 0.135259, throughput: 3 samples/sec
Average loss at batch 14500: 0.145172, throughput: 3 samples/sec
Average loss at batch 14750: 0.146569, throughput: 3 samples/sec
Average loss at batch 15000: 0.131357, throughput: 3 samples/sec
Average loss at batch 15250: 0.132198, throughput: 3 samples/sec
Average loss at batch 15500: 0.135524, throughput: 3 samples/sec
Average loss at batch 15750: 0.132345, throughput: 3 samples/sec
Average loss at batch 16000: 0.138614, throughput: 3 samples/sec
Average loss at batch 16250: 0.140175, throughput: 3 samples/sec
Average loss at batch 16500: 0.145225, throughput: 3 samples/sec
Average loss at batch 16750: 0.141024, throughput: 3 samples/sec
Average loss at batch 17000: 0.129212, throughput: 3 samples/sec
Average loss at batch 17250: 0.137498, throughput: 3 samples/sec
Average loss at batch 17500: 0.131254, throughput: 3 samples/sec
Average loss at batch 17750: 0.137431, throughput: 3 samples/sec
Average loss at batch 18000: 0.140637, throughput: 3 samples/sec
Average loss at batch 18250: 0.140474, throughput: 3 samples/sec
Average loss at batch 18500: 0.141638, throughput: 3 samples/sec
Average loss at batch 18750: 0.132108, throughput: 3 samples/sec
Average loss at batch 19000: 0.137008, throughput: 3 samples/sec
Average loss at batch 19250: 0.133061, throughput: 3 samples/sec
Average loss at batch 19500: 0.133036, throughput: 3 samples/sec
Average loss at batch 19750: 0.141949, throughput: 3 samples/sec
Average loss at batch 20000: 0.139936, throughput: 3 samples/sec
Average loss at batch 20250: 0.146431, throughput: 3 samples/sec
Average loss at batch 20500: 0.134104, throughput: 3 samples/sec
Average loss at batch 20750: 0.132237, throughput: 3 samples/sec
Average loss at batch 21000: 0.137539, throughput: 3 samples/sec
Average loss at batch 21250: 0.131193, throughput: 3 samples/sec
Average loss at batch 21500: 0.136306, throughput: 3 samples/sec
Average loss at batch 21750: 0.143035, throughput: 3 samples/sec
Average loss at batch 22000: 0.144969, throughput: 3 samples/sec
Finished 100 epochs
Evaluation time: 0h0m21s
Accuracy after 100 epochs: 0.09649
Saved after 100 epochs in: model/test_dataset-model/model_iter100
Average loss at batch 22250: 0.136871, throughput: 2 samples/sec
Average loss at batch 22500: 0.131822, throughput: 3 samples/sec
Average loss at batch 22750: 0.137730, throughput: 3 samples/sec
Average loss at batch 23000: 0.129949, throughput: 3 samples/sec
Average loss at batch 23250: 0.141296, throughput: 3 samples/sec
Average loss at batch 23500: 0.134622, throughput: 3 samples/sec
Average loss at batch 23750: 0.141059, throughput: 3 samples/sec
Average loss at batch 24000: 0.145369, throughput: 3 samples/sec
Average loss at batch 24250: 0.130758, throughput: 3 samples/sec
Average loss at batch 24500: 0.135738, throughput: 3 samples/sec
Average loss at batch 24750: 0.134645, throughput: 3 samples/sec
Average loss at batch 25000: 0.136356, throughput: 3 samples/sec
Average loss at batch 25250: 0.138851, throughput: 3 samples/sec
Average loss at batch 25500: 0.138119, throughput: 3 samples/sec
Average loss at batch 25750: 0.144956, throughput: 3 samples/sec
Average loss at batch 26000: 0.136848, throughput: 3 samples/sec
Average loss at batch 26250: 0.134897, throughput: 3 samples/sec
Average loss at batch 26500: 0.135572, throughput: 3 samples/sec
Average loss at batch 26750: 0.130219, throughput: 3 samples/sec
Average loss at batch 27000: 0.138906, throughput: 3 samples/sec
Average loss at batch 27250: 0.142824, throughput: 3 samples/sec
Average loss at batch 27500: 0.140605, throughput: 3 samples/sec
Average loss at batch 27750: 0.139245, throughput: 3 samples/sec
Average loss at batch 28000: 0.130321, throughput: 3 samples/sec
Average loss at batch 28250: 0.140632, throughput: 3 samples/sec
Average loss at batch 28500: 0.128969, throughput: 3 samples/sec
Average loss at batch 28750: 0.139697, throughput: 3 samples/sec
Average loss at batch 29000: 0.135484, throughput: 3 samples/sec
Average loss at batch 29250: 0.143712, throughput: 3 samples/sec
Average loss at batch 29500: 0.146288, throughput: 3 samples/sec
Average loss at batch 29750: 0.127220, throughput: 3 samples/sec
Average loss at batch 30000: 0.137717, throughput: 3 samples/sec
Average loss at batch 30250: 0.131620, throughput: 3 samples/sec
Average loss at batch 30500: 0.136929, throughput: 3 samples/sec
Average loss at batch 30750: 0.137807, throughput: 3 samples/sec
Average loss at batch 31000: 0.143365, throughput: 3 samples/sec
Average loss at batch 31250: 0.148549, throughput: 3 samples/sec
Average loss at batch 31500: 0.129830, throughput: 3 samples/sec
Average loss at batch 31750: 0.133136, throughput: 3 samples/sec
Average loss at batch 32000: 0.137057, throughput: 3 samples/sec
Average loss at batch 32250: 0.131048, throughput: 3 samples/sec
Average loss at batch 32500: 0.137651, throughput: 3 samples/sec
Average loss at batch 32750: 0.141579, throughput: 3 samples/sec
Average loss at batch 33000: 0.141500, throughput: 3 samples/sec
Average loss at batch 33250: 0.141524, throughput: 3 samples/sec
Average loss at batch 33500: 0.129875, throughput: 3 samples/sec
Average loss at batch 33750: 0.139999, throughput: 3 samples/sec
Average loss at batch 34000: 0.126685, throughput: 3 samples/sec
Average loss at batch 34250: 0.141214, throughput: 3 samples/sec
Average loss at batch 34500: 0.140231, throughput: 3 samples/sec
Average loss at batch 34750: 0.140388, throughput: 3 samples/sec
Average loss at batch 35000: 0.143003, throughput: 3 samples/sec
Average loss at batch 35250: 0.131348, throughput: 3 samples/sec
Average loss at batch 35500: 0.137680, throughput: 3 samples/sec
Average loss at batch 35750: 0.129980, throughput: 3 samples/sec
Average loss at batch 36000: 0.138051, throughput: 3 samples/sec
Average loss at batch 36250: 0.138334, throughput: 3 samples/sec
Average loss at batch 36500: 0.141692, throughput: 3 samples/sec
Average loss at batch 36750: 0.146335, throughput: 3 samples/sec
Average loss at batch 37000: 0.132641, throughput: 3 samples/sec
Average loss at batch 37250: 0.132460, throughput: 3 samples/sec
Average loss at batch 37500: 0.135458, throughput: 3 samples/sec
Average loss at batch 37750: 0.132478, throughput: 3 samples/sec
Average loss at batch 38000: 0.136788, throughput: 3 samples/sec
Average loss at batch 38250: 0.142014, throughput: 3 samples/sec
Average loss at batch 38500: 0.142546, throughput: 3 samples/sec
Average loss at batch 38750: 0.139360, throughput: 3 samples/sec
Average loss at batch 39000: 0.130712, throughput: 3 samples/sec
Average loss at batch 39250: 0.141848, throughput: 3 samples/sec
Average loss at batch 39500: 0.129785, throughput: 3 samples/sec
Average loss at batch 39750: 0.134876, throughput: 3 samples/sec
Average loss at batch 40000: 0.142561, throughput: 3 samples/sec
Average loss at batch 40250: 0.139843, throughput: 3 samples/sec
Average loss at batch 40500: 0.146388, throughput: 3 samples/sec
Average loss at batch 40750: 0.128429, throughput: 3 samples/sec
Average loss at batch 41000: 0.137541, throughput: 3 samples/sec
Average loss at batch 41250: 0.132243, throughput: 3 samples/sec
Average loss at batch 41500: 0.136309, throughput: 3 samples/sec
Average loss at batch 41750: 0.139365, throughput: 3 samples/sec
Average loss at batch 42000: 0.142417, throughput: 3 samples/sec
Average loss at batch 42250: 0.139220, throughput: 3 samples/sec
Average loss at batch 42500: 0.137468, throughput: 3 samples/sec
Average loss at batch 42750: 0.133404, throughput: 3 samples/sec
Average loss at batch 43000: 0.136407, throughput: 3 samples/sec
Average loss at batch 43250: 0.130862, throughput: 3 samples/sec
Average loss at batch 43500: 0.138294, throughput: 3 samples/sec
Average loss at batch 43750: 0.142886, throughput: 3 samples/sec
Average loss at batch 44000: 0.143123, throughput: 3 samples/sec
Finished 100 epochs
Evaluation time: 0h0m17s
Accuracy after 200 epochs: 0.09649
Saved after 200 epochs in: model/test_dataset-model/model_iter200
Average loss at batch 44250: 0.138663, throughput: 2 samples/sec
Average loss at batch 44500: 0.131556, throughput: 3 samples/sec
Average loss at batch 44750: 0.136707, throughput: 3 samples/sec
Average loss at batch 45000: 0.131999, throughput: 3 samples/sec
Average loss at batch 45250: 0.136004, throughput: 3 samples/sec
Average loss at batch 45500: 0.137086, throughput: 3 samples/sec
Average loss at batch 45750: 0.142641, throughput: 3 samples/sec
Average loss at batch 46000: 0.145650, throughput: 3 samples/sec
Average loss at batch 46250: 0.129860, throughput: 3 samples/sec
Average loss at batch 46500: 0.137097, throughput: 3 samples/sec
Average loss at batch 46750: 0.133858, throughput: 3 samples/sec
Average loss at batch 47000: 0.136108, throughput: 3 samples/sec
Average loss at batch 47250: 0.134802, throughput: 3 samples/sec
Average loss at batch 47500: 0.143452, throughput: 3 samples/sec
Average loss at batch 47750: 0.148052, throughput: 3 samples/sec
Average loss at batch 48000: 0.133201, throughput: 3 samples/sec
Average loss at batch 48250: 0.132965, throughput: 3 samples/sec
Average loss at batch 48500: 0.135257, throughput: 3 samples/sec
Average loss at batch 48750: 0.132595, throughput: 3 samples/sec
Average loss at batch 49000: 0.137465, throughput: 3 samples/sec
Average loss at batch 49250: 0.140421, throughput: 3 samples/sec
Average loss at batch 49500: 0.143884, throughput: 3 samples/sec
Average loss at batch 49750: 0.136662, throughput: 3 samples/sec
Average loss at batch 50000: 0.133104, throughput: 3 samples/sec
Average loss at batch 50250: 0.140104, throughput: 3 samples/sec
Average loss at batch 50500: 0.128160, throughput: 3 samples/sec
Average loss at batch 50750: 0.140676, throughput: 3 samples/sec
Average loss at batch 51000: 0.137767, throughput: 3 samples/sec
Average loss at batch 51250: 0.141415, throughput: 3 samples/sec
Average loss at batch 51500: 0.145552, throughput: 3 samples/sec
Average loss at batch 51750: 0.128710, throughput: 3 samples/sec
Average loss at batch 52000: 0.136397, throughput: 3 samples/sec
Average loss at batch 52250: 0.133991, throughput: 3 samples/sec
Average loss at batch 52500: 0.134263, throughput: 3 samples/sec
Average loss at batch 52750: 0.138802, throughput: 3 samples/sec
Average loss at batch 53000: 0.140598, throughput: 3 samples/sec
Average loss at batch 53250: 0.146371, throughput: 3 samples/sec
Average loss at batch 53500: 0.134580, throughput: 3 samples/sec
Average loss at batch 53750: 0.133111, throughput: 3 samples/sec
Average loss at batch 54000: 0.136518, throughput: 3 samples/sec
Average loss at batch 54250: 0.131612, throughput: 3 samples/sec
Average loss at batch 54500: 0.136419, throughput: 3 samples/sec
Average loss at batch 54750: 0.140362, throughput: 3 samples/sec
Average loss at batch 55000: 0.144226, throughput: 3 samples/sec
Average loss at batch 55250: 0.141210, throughput: 3 samples/sec
Average loss at batch 55500: 0.128602, throughput: 3 samples/sec
Average loss at batch 55750: 0.139907, throughput: 3 samples/sec
Average loss at batch 56000: 0.130415, throughput: 3 samples/sec
Average loss at batch 56250: 0.136683, throughput: 3 samples/sec
Average loss at batch 56500: 0.140884, throughput: 3 samples/sec
Average loss at batch 56750: 0.139751, throughput: 3 samples/sec
Average loss at batch 57000: 0.148199, throughput: 3 samples/sec
Average loss at batch 57250: 0.128225, throughput: 3 samples/sec
Average loss at batch 57500: 0.137868, throughput: 3 samples/sec
Average loss at batch 57750: 0.131920, throughput: 3 samples/sec
Average loss at batch 58000: 0.134923, throughput: 3 samples/sec
Average loss at batch 58250: 0.139174, throughput: 3 samples/sec
Average loss at batch 58500: 0.140582, throughput: 3 samples/sec
Average loss at batch 58750: 0.147675, throughput: 3 samples/sec
Average loss at batch 59000: 0.131522, throughput: 3 samples/sec
Average loss at batch 59250: 0.135096, throughput: 3 samples/sec
Average loss at batch 59500: 0.134853, throughput: 3 samples/sec
Average loss at batch 59750: 0.131238, throughput: 3 samples/sec
Average loss at batch 60000: 0.138819, throughput: 3 samples/sec
Average loss at batch 60250: 0.140366, throughput: 3 samples/sec
Average loss at batch 60500: 0.138972, throughput: 3 samples/sec
Average loss at batch 60750: 0.144431, throughput: 3 samples/sec
Average loss at batch 61000: 0.131601, throughput: 3 samples/sec
Average loss at batch 61250: 0.138038, throughput: 3 samples/sec
Average loss at batch 61500: 0.130192, throughput: 3 samples/sec
Average loss at batch 61750: 0.138295, throughput: 3 samples/sec
Average loss at batch 62000: 0.139771, throughput: 3 samples/sec
Average loss at batch 62250: 0.141297, throughput: 3 samples/sec
Average loss at batch 62500: 0.145646, throughput: 3 samples/sec
Average loss at batch 62750: 0.128461, throughput: 3 samples/sec
Average loss at batch 63000: 0.134020, throughput: 3 samples/sec
Average loss at batch 63250: 0.135197, throughput: 3 samples/sec
Average loss at batch 63500: 0.135664, throughput: 3 samples/sec
Average loss at batch 63750: 0.136685, throughput: 3 samples/sec
Average loss at batch 64000: 0.140934, throughput: 3 samples/sec
Average loss at batch 64250: 0.145585, throughput: 3 samples/sec
Average loss at batch 64500: 0.135877, throughput: 3 samples/sec
Average loss at batch 64750: 0.134279, throughput: 3 samples/sec
Average loss at batch 65000: 0.136065, throughput: 3 samples/sec
Average loss at batch 65250: 0.129985, throughput: 3 samples/sec
Average loss at batch 65500: 0.138081, throughput: 3 samples/sec
Average loss at batch 65750: 0.141547, throughput: 3 samples/sec
Average loss at batch 66000: 0.145466, throughput: 3 samples/sec
Finished 100 epochs
Evaluation time: 0h0m17s
Accuracy after 300 epochs: 0.09649
Average loss at batch 66250: 0.137880, throughput: 2 samples/sec
Average loss at batch 66500: 0.130645, throughput: 3 samples/sec
Average loss at batch 66750: 0.137175, throughput: 3 samples/sec
Average loss at batch 67000: 0.131852, throughput: 3 samples/sec
Average loss at batch 67250: 0.138747, throughput: 3 samples/sec
Average loss at batch 67500: 0.139100, throughput: 3 samples/sec
Average loss at batch 67750: 0.140023, throughput: 3 samples/sec
Average loss at batch 68000: 0.147478, throughput: 3 samples/sec
Average loss at batch 68250: 0.127633, throughput: 3 samples/sec
Average loss at batch 68500: 0.137544, throughput: 3 samples/sec
Average loss at batch 68750: 0.132890, throughput: 3 samples/sec
Average loss at batch 69000: 0.134576, throughput: 3 samples/sec
Average loss at batch 69250: 0.137544, throughput: 3 samples/sec
Average loss at batch 69500: 0.141237, throughput: 3 samples/sec
Average loss at batch 69750: 0.146454, throughput: 3 samples/sec
Average loss at batch 70000: 0.135505, throughput: 3 samples/sec
Average loss at batch 70250: 0.132883, throughput: 3 samples/sec
Average loss at batch 70500: 0.135282, throughput: 3 samples/sec
Average loss at batch 70750: 0.132270, throughput: 3 samples/sec
Average loss at batch 71000: 0.136325, throughput: 3 samples/sec
Average loss at batch 71250: 0.142149, throughput: 3 samples/sec
Average loss at batch 71500: 0.139003, throughput: 3 samples/sec
Average loss at batch 71750: 0.145836, throughput: 3 samples/sec
Average loss at batch 72000: 0.130294, throughput: 3 samples/sec
Average loss at batch 72250: 0.135616, throughput: 3 samples/sec
Average loss at batch 72500: 0.130151, throughput: 3 samples/sec
Average loss at batch 72750: 0.139301, throughput: 3 samples/sec
Average loss at batch 73000: 0.139303, throughput: 3 samples/sec
Average loss at batch 73250: 0.142394, throughput: 3 samples/sec
Average loss at batch 73500: 0.146361, throughput: 3 samples/sec
Average loss at batch 73750: 0.127902, throughput: 3 samples/sec
Average loss at batch 74000: 0.136160, throughput: 3 samples/sec
Average loss at batch 74250: 0.132233, throughput: 3 samples/sec
Average loss at batch 74500: 0.135856, throughput: 3 samples/sec
Average loss at batch 74750: 0.136132, throughput: 3 samples/sec
Average loss at batch 75000: 0.143963, throughput: 3 samples/sec
Average loss at batch 75250: 0.150061, throughput: 3 samples/sec
Average loss at batch 75500: 0.130918, throughput: 3 samples/sec
Average loss at batch 75750: 0.132581, throughput: 3 samples/sec
Average loss at batch 76000: 0.136975, throughput: 3 samples/sec
Average loss at batch 76250: 0.127915, throughput: 3 samples/sec
Average loss at batch 76500: 0.141240, throughput: 3 samples/sec
Average loss at batch 76750: 0.140178, throughput: 3 samples/sec
Average loss at batch 77000: 0.143170, throughput: 3 samples/sec
Average loss at batch 77250: 0.140554, throughput: 3 samples/sec
Average loss at batch 77500: 0.130553, throughput: 3 samples/sec
Average loss at batch 77750: 0.138200, throughput: 3 samples/sec
Average loss at batch 78000: 0.129593, throughput: 3 samples/sec
Average loss at batch 78250: 0.138101, throughput: 3 samples/sec
Average loss at batch 78500: 0.138474, throughput: 3 samples/sec
Average loss at batch 78750: 0.142008, throughput: 3 samples/sec
Average loss at batch 79000: 0.148007, throughput: 3 samples/sec
Average loss at batch 79250: 0.127454, throughput: 3 samples/sec
Average loss at batch 79500: 0.137597, throughput: 3 samples/sec
Average loss at batch 79750: 0.132156, throughput: 3 samples/sec
Average loss at batch 80000: 0.135956, throughput: 3 samples/sec
Average loss at batch 80250: 0.136692, throughput: 3 samples/sec
Average loss at batch 80500: 0.142313, throughput: 3 samples/sec
Average loss at batch 80750: 0.147101, throughput: 3 samples/sec
Average loss at batch 81000: 0.133842, throughput: 3 samples/sec
Average loss at batch 81250: 0.132121, throughput: 3 samples/sec
Average loss at batch 81500: 0.137256, throughput: 3 samples/sec
Average loss at batch 81750: 0.128879, throughput: 3 samples/sec
Average loss at batch 82000: 0.140973, throughput: 3 samples/sec
Average loss at batch 82250: 0.138297, throughput: 3 samples/sec
Average loss at batch 82500: 0.143386, throughput: 3 samples/sec
Average loss at batch 82750: 0.140194, throughput: 3 samples/sec
Average loss at batch 83000: 0.130981, throughput: 3 samples/sec
Average loss at batch 83250: 0.138324, throughput: 3 samples/sec
Average loss at batch 83500: 0.130608, throughput: 3 samples/sec
Average loss at batch 83750: 0.139397, throughput: 3 samples/sec
Average loss at batch 84000: 0.140787, throughput: 3 samples/sec
Average loss at batch 84250: 0.137898, throughput: 3 samples/sec
Average loss at batch 84500: 0.144381, throughput: 3 samples/sec
Average loss at batch 84750: 0.132528, throughput: 3 samples/sec
Average loss at batch 85000: 0.136025, throughput: 3 samples/sec
Average loss at batch 85250: 0.132634, throughput: 3 samples/sec
Average loss at batch 85500: 0.135715, throughput: 3 samples/sec
Average loss at batch 85750: 0.140010, throughput: 3 samples/sec
Average loss at batch 86000: 0.139636, throughput: 3 samples/sec
Average loss at batch 86250: 0.147105, throughput: 3 samples/sec
Average loss at batch 86500: 0.132856, throughput: 3 samples/sec
Average loss at batch 86750: 0.132727, throughput: 3 samples/sec
Average loss at batch 87000: 0.138209, throughput: 3 samples/sec
Average loss at batch 87250: 0.129693, throughput: 3 samples/sec
Average loss at batch 87500: 0.138898, throughput: 3 samples/sec
Average loss at batch 87750: 0.141063, throughput: 3 samples/sec
Average loss at batch 88000: 0.143555, throughput: 3 samples/sec
Finished 100 epochs
Evaluation time: 0h0m17s
Accuracy after 400 epochs: 0.09649
Not improved for 200 epochs, stopping training
Best scores - epoch 200:
Loss: 0.09649
Evaluation time: 0h0m17s
Loss: 0.09648507379644183

[urialon]

Can you try to do the same, but point the flag '--test' to the training file?
I.e., training and testing on the training file to see if the model was able to memorize it.

[lishi0927]

Thank you for your advice.
I have another question, you split code tokens into subtokens, for example you will split the ArrayList into Array and List; But if we give codes with other temporary names, the result will be different?
The token representations will have different names, but with the same paths, the results will be different?
The same token representations in different codes with different functions or types will influence the final results?

[urialon]

Yes,
you can try this in the online demo at code2seq.org

Best,
Uri

[lishi0927]

Thank you.
I have tried your demo.
for example,
Example 9 in your online demo
void f() { boolean done = false; while (!done) { if (remaining() <= 0) { done = true; } } }
This java codes predict "check" and "done"
void f() { boolean target = false; while (!target) { if (remaining() <= 0) { target = true; } } }
But this java codes give "skip", "to', "first";
The difference between these two java codes is only one variable name, we use "target" instead of "done".

[urialon]

This is correct.
The model learns to use variable names for its prediction. As a consequence, the model also depends on "the right" variable names. If you want a model that is 100% "robust" to variable names, you can train a model from scratch without any variable names.

We have recently published another paper that shows how to adversarially choose variable names such that the model will output a prediction of our desire. See:
https://arxiv.org/pdf/1910.07517.pdf

[lishi0927]

Okay, Wonderful work! I'll read this paper.
Thanks for a series of research work about this topic.
I'm interested about the code to vector, and I have another confusion about using network to encode program, can these encoded vectors decode to a program?

[urialon]

We have another recently published paper that gets a program with a "hole", and generates the missing piece by treating the missing piece as a sub-tree, and predicting this sub-tree node-by-node:

https://arxiv.org/pdf/1910.00577.pdf

(the source code for this paper is not public yet, hopefully in a month or two)

[lishi0927]

Thank you. I'll read these two paper.

[urialon]

Cool, let me know if you have any questions!

[lishi0927]

Okay.
Thank you for your patience.

[lishi0927]

I have read your these two papers. It's really interesting.
And I have another question, we know the programs are used to compute some results with given inputs, can these embedding vector in the neural network still retain these attributes? Then we could know two codes are the same because all inputs can produce the same results, or with the input, the embedding vector can still output the right results?

[urialon]

This is a very interesting question.
This question, in general, is undecidable.
However, I do think that what you're suggesting is possible using a learning model, up to some accuracy.
The main problem is that we don't have a good dataset to test this.

[lishi0927]

Thank you.
I have another question about this code.
I know in neural network model, the function "compute_contexts" can encode the (source, path, target), but i find my training loss has a vibration. because we set the "random_contexts = True". I'm curious about "random_contexts" for the same file will receive same embedding vectors returned by "compute_contexts" function or not?

[urialon]

At test time - random_contexts is disabled, so testing is deterministic, and the same file will get the same values.
If random_contexts=True, it only affects the training.

[lishi0927]

Okay, I get it.
Have you ever done this experiment? the random contexts will produce same embedding vector or not?

[urialon]

Of course. I haven't checked the exact vector values, but you can verify that loading a trained mode and testing it multiple times - always produces the exact same F1 score and the same 'log.txt' file.
If there was an indeterminism at test time, the results would have been slightly different every test run.
(I'm talking about testing only, while always taking the same trained model)

[lishi0927]

Yes, I get it. The trained model will change training parameters then the embedding vector will not be static.
The random contexts gives some different inputs and then my training loss has a vibration.
Therefore, the training step needs to make these different inputs output a static embedding vector;

[urialon]

I'm not sure I understood your last comment.

In any case, you can try to turn random_contexts to False and see if it matters.
I believe that the main cause of your training difficulties is that your dataset has only 40 examples. 40 training examples can rarely allow any learning.

[lishi0927]

Okay, Thank you for your advice.

[lishi0927]

I have tried some experiments about the above question.
I use "models/java-large-model/model_iter52.release" pretrained model, I change a few "reader.py" codes, which delete ifstatement "not self.is_evaluating". Then "random_contexts=True" can affect the testing step.
I use this example 7 in your online demo:
int f(Object target) { int i = 0; for (Object elem: this.elements) { if (elem.equals(target)) { return i; } i++; } return -1; }
I use this command python3 code2seq.py --load models/java-large-model/model_iter52.release --predict and output batched_contexts and path_strings.
I run the command twice and output are shown as follows;
I find the batch_contexts and path strings are different, but the predicted results are the same.

batch_contexts 1:
[[[ 0.35196486 -0.30670446 0.59376854 ... 0.16907062 0.1261587
-0.12858267]
[ 0.4417759 -0.4660875 0.2712744 ... 0.25732806 0.41818997
0.16799302]
[ 0. 0. 0. ... 0. 0.
0. ]
...
[ 0. 0. 0. ... 0. 0.
0. ]
[ 0.15033087 0.52494913 0.12702663 ... -0.72016996 0.7227744
0.06895924]
[-0.01451584 -0.57259256 -0.15304852 ... 0.5781429 -0.41368
-0.57052255]]]
path string 1:
[[[b'Prim0|VDE|Ex|Bk|Foreach|Bk|If|Cal0|Nm0']
[b'This0|Fld|Foreach|Bk|If|Cal0|Nm0']
[b'']
[b'IntEx1|VD|VDE|Ex|Bk|Foreach|VDE|VD|VDID0']
[b'']
[b'Cls0|VDE|Foreach|Bk|If|Cal0|Nm3']
[b'Nm2|Cal|If|Bk|Ret|Nm0']
[b'']
[b'VDID0|Prm|Mth|Bk|Ex|VDE|Prim0']
[b'Nm2|Cal|If|Bk|Foreach|Bk|Ret|Neg|IntEx0']
[b'Prim0|VDE|Ex|Bk|Foreach|Bk|If|Cal0|Nm3']
[b'IntEx1|VD|VDE|Ex|Bk|Foreach|VDE|Cls0']
[b'']
[b'VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm0']
[b'Nm1|Mth|Bk|Ex|VDE|VD|VDID0']
[b'']
[b'']
[b'']
[b'']
[b'Nm2|Fld|Foreach|Bk|Ex|PosInc|Nm0']
[b'Nm1|Mth|Prm|VDID0']
[b'Cls1|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm0']
[b'']
[b'']
[b'']
[b'Nm3|Cal|If|Bk|Ret|Nm0']
[b'']
[b'VDID0|Prm|Mth|Bk|Ex|VDE|VD|IntEx1']
[b'Cls1|Prm|Mth|Bk|Ex|VDE|VD|IntEx1']
[b'VDID0|VD|VDE|Foreach|Bk|If|Cal0|Nm2']
[b'Nm0|Cal|If|Bk|Ex|PosInc|Nm0']
[b'']
[b'Nm0|Cal|Nm2']
[b'Cls1|Prm|Mth|Bk|Ret|Neg|IntEx0']
[b'']
[b'This0|Fld|Foreach|Bk|If|Cal0|Nm2']
[b'']
[b'']
[b'VDID0|Prm|Mth|Bk|Foreach|VDE|Cls0']
[b'']
[b'Prim0|VDE|Ex|Bk|Foreach|Fld1|This0']
[b'']
[b'']
[b'']
[b'']
[b'']
[b'']
[b'']
[b'']
[b'Cls1|Prm|Mth|Bk|Foreach|Bk|Ex|PosInc|Nm0']
[b'']
[b'']
[b'Cls1|Prm|Mth|Bk|Foreach|Fld1|This0']
[b'']
[b'Nm2|Fld|Foreach|Bk|If|Bk|Ret|Nm0']
[b'']
[b'']
[b'This0|Fld|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'Cls1|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm3']
[b'']
[b'']
[b'Cls0|VDE|Foreach|Bk|If|Cal0|Nm2']
[b'']
[b'']
[b'Nm1|Mth|Bk|Foreach|Bk|If|Bk|Ret|Nm0']
[b'']
[b'']
[b'Nm2|Cal|Nm3']
[b'Nm0|Cal|If|Bk|Foreach|Bk|Ret|Neg|IntEx0']
[b'Cls1|Prm|Mth|Bk|Foreach|Fld1|Nm2']
[b'']
[b'Nm2|Fld|Foreach|Bk|If|Cal0|Nm2']
[b'']
[b'Prim0|VDE|Ex|Bk|Foreach|VDE|Cls0']
[b'']
[b'']
[b'Nm1|Mth|Bk|Ex|VDE|Prim0']
[b'']
[b'VDID0|VD|VDE|Ex|Bk|Foreach|VDE|Cls0']
[b'']
[b'']
[b'']
[b'Prim0|Mth|Prm|VDID0']
[b'']
[b'Nm1|Mth|Bk|Ex|VDE|VD|IntEx1']
[b'']
[b'']
[b'VDID0|VD|VDE|Foreach|Bk|Ex|PosInc|Nm0']
[b'VDID0|Prm|Mth|Bk|Ret|Neg|IntEx0']
[b'Prim0|VDE|Ex|Bk|Foreach|Bk|If|Cal0|Nm2']
[b'Nm3|Cal|If|Bk|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'']
[b'']
[b'Cls1|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm2']
[b'']
[b'']
[b'Nm1|Mth|Bk|Foreach|Fld1|This0']
[b'Nm0|Cal|If|Bk|Ret|Nm0']
[b'Cls1|Prm|Mth|Bk|Ex|VDE|Prim0']
[b'']
[b'Nm1|Mth|Bk|Foreach|Bk|If|Cal0|Nm0']
[b'Cls0|VDE|Foreach|Bk|If|Cal0|Nm0']
[b'Nm2|Cal|If|Bk|Ex|PosInc|Nm0']
[b'Prim0|VDE|Ex|Bk|Foreach|Bk|Ex|PosInc|Nm0']
[b'']
[b'']
[b'']
[b'IntEx1|VD|VDE|Ex|Bk|Foreach|Fld1|This0']
[b'Prim0|VDE|Ex|Bk|Ret|Neg|IntEx0']
[b'']
[b'Nm2|Fld|Foreach|Bk|If|Cal0|Nm0']
[b'']
[b'VDID0|VD|VDE|Foreach|Fld1|Nm2']
[b'Nm1|Mth|Prm|Cls1']
[b'']
[b'Nm3|Cal|If|Bk|Ex|PosInc|Nm0']
[b'']
[b'']
[b'Cls1|Prm|Mth|Bk|Foreach|VDE|VD|VDID0']
[b'Prim0|Mth|Nm1']
[b'VDID0|VD|VDE|Foreach|Bk|If|Cal0|Nm0']
[b'Cls0|VDE|Foreach|Fld1|Nm2']
[b'IntEx1|VD|VDE|Ex|Bk|Ret|Neg|IntEx0']
[b'']
[b'']
[b'VDID0|VD|VDE|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'This0|Fld|Foreach|Bk|If|Bk|Ret|Nm0']
[b'']
[b'']
[b'This0|Fld|Foreach|Bk|Ex|PosInc|Nm0']
[b'VDID0|VD|VDE|Foreach|Bk|If|Cal0|Nm3']
[b'']
[b'']
[b'Nm0|Ret|Bk|If|Bk|Ex|PosInc|Nm0']
[b'']
[b'VDID0|VD|VDE|Ex|Bk|Foreach|Fld1|Nm2']
[b'Nm2|Fld|Foreach|Bk|If|Cal0|Nm3']
[b'VDID0|VD|VDE|Ex|Bk|Ret|Neg|IntEx0']
[b'VDID0|Prm|Mth|Bk|Foreach|VDE|VD|VDID0']
[b'This0|Fld|Nm2']
[b'Prim0|VDE|Ex|Bk|Foreach|VDE|VD|VDID0']
[b'IntEx1|VD|VDE|Ex|Bk|Foreach|Fld1|Nm2']
[b'Nm0|PosInc|Ex|Bk|Foreach|Bk|Ret|Neg|IntEx0']
[b'Nm1|Mth|Bk|Foreach|VDE|Cls0']
[b'']
[b'']
[b'Prim0|VDE|VD|VDID0']
[b'Nm1|Mth|Bk|Foreach|Bk|If|Cal0|Nm3']
[b'']
[b'VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm3']
[b'Nm1|Mth|Bk|Ret|Neg|IntEx0']
[b'']
[b'']
[b'']
[b'Prim0|VDE|VD|IntEx1']
[b'VDID0|Prm|Mth|Bk|Ex|VDE|VD|VDID0']
[b'VDID0|VD|VDE|Ex|Bk|Foreach|Fld1|This0']
[b'']
[b'Cls1|Prm|Mth|Bk|Ex|VDE|VD|VDID0']
[b'Cls0|VDE|Foreach|Bk|Ex|PosInc|Nm0']
[b'']
[b'Cls0|VDE|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'']
[b'VDID0|VD|VDE|Ex|Bk|Foreach|VDE|VD|VDID0']
[b'']
[b'']
[b'Cls1|Prm|Mth|Bk|Foreach|VDE|Cls0']
[b'Nm1|Mth|Bk|Foreach|Bk|Ex|PosInc|Nm0']
[b'VDID0|Prm|Cls1']
[b'Prim0|Mth|Prm|Cls1']
[b'Prim0|VDE|Ex|Bk|Foreach|Fld1|Nm2']
[b'Nm2|Fld|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'Nm1|Mth|Bk|Foreach|Fld1|Nm2']
[b'']
[b'VDID0|VD|VDE|Foreach|Bk|If|Bk|Ret|Nm0']
[b'VDID0|VD|VDE|Foreach|Fld1|This0']
[b'']
[b'']
[b'']
[b'VDID0|VD|IntEx1']
[b'Cls0|VDE|VD|VDID0']
[b'']
[b'Cls0|VDE|Foreach|Bk|If|Bk|Ret|Nm0']
[b'']
[b'Nm1|Mth|Bk|Foreach|Bk|If|Cal0|Nm2']
[b'Cls0|VDE|Foreach|Fld1|This0']
[b'VDID0|Prm|Mth|Bk|Foreach|Fld1|Nm2']
[b'']
[b'VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm2']
[b'VDID0|Prm|Mth|Bk|Foreach|Bk|Ex|PosInc|Nm0']
[b'VDID0|Prm|Mth|Bk|Foreach|Fld1|This0']
[b'']
[b'']
[b'This0|Fld|Foreach|Bk|If|Cal0|Nm3']
[b'Nm1|Mth|Bk|Foreach|VDE|VD|VDID0']]]

result 1:
Predicted: ['index', 'of', 'target']
Attention:
TIMESTEP: 0 : index
0.106182 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.094745 context: int,(PrimitiveType0)^(MethodDeclaration)(NameExpr1),METHOD_NAME
0.071685 context: object,(ClassOrInterfaceType1)^(Parameter)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.062928 context: int,(PrimitiveType0)^(MethodDeclaration)(Parameter)(VariableDeclaratorId0),target
0.045155 context: i,(NameExpr0)^(UnaryExpr:posIncrement)^(ExpressionStmt)^(BlockStmt)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.044015 context: equals,(NameExpr3)^(MethodCallExpr)^(IfStmt)^(BlockStmt)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.042547 context: METHOD_NAME,(NameExpr1)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.039831 context: int,(PrimitiveType0)^(VariableDeclarationExpr)^(ExpressionStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.036678 context: target,(NameExpr2)^(MethodCallExpr)^(IfStmt)^(BlockStmt)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.036100 context: this,(ThisExpr0)^(FieldAccessExpr)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
Attention:
TIMESTEP: 1 : of
0.068563 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.054894 context: object,(ClassOrInterfaceType1)^(Parameter)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.039301 context: int,(PrimitiveType0)^(MethodDeclaration)(NameExpr1),METHOD_NAME
0.032482 context: object,(ClassOrInterfaceType1)^(Parameter)^(MethodDeclaration)(BlockStmt)(ForeachStmt)(FieldAccessExpr1)(ThisExpr0),this
0.031575 context: i,(NameExpr0)^(UnaryExpr:posIncrement)^(ExpressionStmt)^(BlockStmt)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.030011 context: METHOD_NAME,(NameExpr1)^(MethodDeclaration)(BlockStmt)(ForeachStmt)(BlockStmt)(IfStmt)(MethodCallExpr0)(NameExpr3),equals
0.027228 context: elem,(NameExpr0)^(MethodCallExpr)^(IfStmt)^(BlockStmt)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.025339 context: elem,(VariableDeclaratorId0)^(VariableDeclarator)^(VariableDeclarationExpr)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.024614 context: this,(ThisExpr0)^(FieldAccessExpr)^(ForeachStmt)(BlockStmt)(IfStmt)(MethodCallExpr0)(NameExpr3),equals
0.024533 context: METHOD_NAME,(NameExpr1)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
Attention:
TIMESTEP: 2 : target
0.142251 context: METHOD_NAME,(NameExpr1)^(MethodDeclaration)(Parameter)(VariableDeclaratorId0),target
0.066175 context: target,(VariableDeclaratorId0)^(Parameter)(ClassOrInterfaceType1),object
0.053913 context: int,(PrimitiveType0)^(MethodDeclaration)(Parameter)(VariableDeclaratorId0),target
0.049049 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ForeachStmt)(FieldAccessExpr1)(ThisExpr0),this
0.041023 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ForeachStmt)(BlockStmt)(IfStmt)(MethodCallExpr0)(NameExpr0),elem
0.033182 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ExpressionStmt)(VariableDeclarationExpr)(PrimitiveType0),int
0.030040 context: METHOD_NAME,(NameExpr1)^(MethodDeclaration)(Parameter)(ClassOrInterfaceType1),object
0.029911 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ForeachStmt)(BlockStmt)(IfStmt)(MethodCallExpr0)(NameExpr2),target
0.023267 context: elem,(NameExpr0)^(MethodCallExpr)(NameExpr2),target
0.022454 context: int,(PrimitiveType0)^(MethodDeclaration)(Parameter)(ClassOrInterfaceType1),object

batch_contexts 2:
[[[ 2.30489016e-01 1.21090874e-01 9.23644900e-02 ... -2.71725416e-01
-5.76230884e-01 -3.91485155e-01]
[ 2.49215409e-01 4.13360268e-01 -4.24404025e-01 ... 1.32337213e-04
4.97161597e-02 3.15663069e-01]
[ 7.86302388e-01 -4.95621920e-01 9.74471271e-01 ... 3.99053812e-01
-2.86332995e-01 -2.44823828e-01]
...
[ 4.41775888e-01 -4.66087490e-01 2.71274388e-01 ... 2.57328063e-01
4.18189973e-01 1.67993024e-01]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 ... 0.00000000e+00
0.00000000e+00 0.00000000e+00]
[ 0.00000000e+00 0.00000000e+00 0.00000000e+00 ... 0.00000000e+00
0.00000000e+00 0.00000000e+00]]]

path string 2:
[[[b'Prim0|Mth|Nm1']
[b'VDID0|Prm|Mth|Bk|Ex|VDE|VD|VDID0']
[b'VDID0|VD|VDE|Foreach|Bk|If|Bk|Ret|Nm0']
[b'Nm0|Cal|If|Bk|Ex|PosInc|Nm0']
[b'']
[b'Nm1|Mth|Bk|Ex|VDE|VD|VDID0']
[b'VDID0|VD|VDE|Foreach|Bk|If|Cal0|Nm3']
[b'Nm0|Cal|Nm2']
[b'IntEx1|VD|VDE|Ex|Bk|Ret|Neg|IntEx0']
[b'Prim0|VDE|Ex|Bk|Foreach|Bk|Ex|PosInc|Nm0']
[b'Nm2|Fld|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'Cls1|Prm|Mth|Bk|Foreach|Fld1|Nm2']
[b'']
[b'']
[b'']
[b'Nm1|Mth|Bk|Foreach|Fld1|This0']
[b'VDID0|VD|VDE|Foreach|Fld1|Nm2']
[b'Nm2|Fld|Foreach|Bk|If|Cal0|Nm0']
[b'Cls1|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm2']
[b'']
[b'VDID0|VD|VDE|Ex|Bk|Foreach|VDE|VD|VDID0']
[b'']
[b'']
[b'Cls1|Prm|Mth|Bk|Ex|VDE|Prim0']
[b'']
[b'']
[b'Cls1|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm3']
[b'Cls1|Prm|Mth|Bk|Ex|VDE|VD|IntEx1']
[b'VDID0|Prm|Mth|Bk|Foreach|Fld1|This0']
[b'Cls0|VDE|VD|VDID0']
[b'This0|Fld|Foreach|Bk|If|Cal0|Nm2']
[b'VDID0|VD|VDE|Foreach|Bk|Ex|PosInc|Nm0']
[b'Nm2|Cal|Nm3']
[b'']
[b'Cls0|VDE|Foreach|Bk|Ex|PosInc|Nm0']
[b'']
[b'VDID0|VD|VDE|Ex|Bk|Foreach|VDE|Cls0']
[b'']
[b'']
[b'']
[b'']
[b'Prim0|Mth|Prm|VDID0']
[b'Prim0|VDE|Ex|Bk|Foreach|VDE|VD|VDID0']
[b'']
[b'']
[b'VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm0']
[b'']
[b'']
[b'This0|Fld|Foreach|Bk|If|Bk|Ret|Nm0']
[b'Nm0|Ret|Bk|If|Bk|Ex|PosInc|Nm0']
[b'']
[b'Cls1|Prm|Mth|Bk|Foreach|Fld1|This0']
[b'Nm1|Mth|Bk|Foreach|Bk|If|Bk|Ret|Nm0']
[b'VDID0|Prm|Mth|Bk|Ex|VDE|VD|IntEx1']
[b'VDID0|Prm|Mth|Bk|Ret|Neg|IntEx0']
[b'This0|Fld|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'']
[b'']
[b'IntEx1|VD|VDE|Ex|Bk|Foreach|VDE|VD|VDID0']
[b'']
[b'']
[b'']
[b'Cls0|VDE|Foreach|Fld1|This0']
[b'']
[b'Cls0|VDE|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'']
[b'']
[b'']
[b'This0|Fld|Foreach|Bk|Ex|PosInc|Nm0']
[b'']
[b'']
[b'Nm1|Mth|Bk|Foreach|Bk|Ex|PosInc|Nm0']
[b'VDID0|Prm|Mth|Bk|Ex|VDE|Prim0']
[b'']
[b'Cls0|VDE|Foreach|Bk|If|Cal0|Nm3']
[b'Prim0|VDE|Ex|Bk|Ret|Neg|IntEx0']
[b'']
[b'']
[b'']
[b'']
[b'']
[b'Cls1|Prm|Mth|Bk|Foreach|Bk|Ex|PosInc|Nm0']
[b'Prim0|VDE|VD|VDID0']
[b'VDID0|VD|VDE|Foreach|Bk|If|Cal0|Nm0']
[b'']
[b'']
[b'IntEx1|VD|VDE|Ex|Bk|Foreach|VDE|Cls0']
[b'Prim0|VDE|Ex|Bk|Foreach|Bk|If|Cal0|Nm3']
[b'']
[b'Prim0|VDE|Ex|Bk|Foreach|Bk|If|Cal0|Nm2']
[b'VDID0|VD|VDE|Foreach|Bk|Ret|Neg|IntEx0']
[b'Nm2|Cal|If|Bk|Ret|Nm0']
[b'This0|Fld|Foreach|Bk|If|Cal0|Nm3']
[b'']
[b'']
[b'']
[b'']
[b'']
[b'Cls0|VDE|Foreach|Bk|If|Cal0|Nm2']
[b'']
[b'']
[b'Nm2|Fld|Foreach|Bk|If|Bk|Ret|Nm0']
[b'VDID0|VD|VDE|Ex|Bk|Foreach|Fld1|Nm2']
[b'VDID0|Prm|Mth|Bk|Foreach|VDE|VD|VDID0']
[b'Nm1|Mth|Bk|Ex|VDE|Prim0']
[b'']
[b'Nm1|Mth|Bk|Foreach|Bk|If|Cal0|Nm3']
[b'Nm0|PosInc|Ex|Bk|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'Nm3|Cal|If|Bk|Ret|Nm0']
[b'VDID0|Prm|Mth|Bk|Foreach|Bk|Ex|PosInc|Nm0']
[b'Nm2|Cal|If|Bk|Foreach|Bk|Ret|Neg|IntEx0']
[b'Nm1|Mth|Prm|VDID0']
[b'Prim0|VDE|Ex|Bk|Foreach|Bk|If|Cal0|Nm0']
[b'VDID0|VD|VDE|Foreach|Bk|If|Cal0|Nm2']
[b'']
[b'Prim0|VDE|Ex|Bk|Foreach|Fld1|This0']
[b'Prim0|VDE|Ex|Bk|Foreach|Fld1|Nm2']
[b'Nm1|Mth|Bk|Foreach|Bk|If|Cal0|Nm2']
[b'Cls1|Prm|Mth|Bk|Foreach|VDE|Cls0']
[b'']
[b'']
[b'Nm2|Fld|Foreach|Bk|If|Cal0|Nm2']
[b'']
[b'Nm1|Mth|Bk|Ex|VDE|VD|IntEx1']
[b'']
[b'VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm2']
[b'']
[b'']
[b'Cls1|Prm|Mth|Bk|Ex|VDE|VD|VDID0']
[b'Nm1|Mth|Prm|Cls1']
[b'']
[b'Nm2|Cal|If|Bk|Ex|PosInc|Nm0']
[b'VDID0|Prm|Cls1']
[b'Nm2|Fld|Foreach|Bk|Ex|PosInc|Nm0']
[b'Prim0|VDE|Ex|Bk|Foreach|VDE|Cls0']
[b'']
[b'Nm2|Fld|Foreach|Bk|If|Cal0|Nm3']
[b'']
[b'']
[b'']
[b'Nm1|Mth|Bk|Ret|Neg|IntEx0']
[b'Prim0|VDE|VD|IntEx1']
[b'']
[b'']
[b'VDID0|VD|VDE|Ex|Bk|Ret|Neg|IntEx0']
[b'']
[b'']
[b'VDID0|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm3']
[b'']
[b'Nm1|Mth|Bk|Foreach|VDE|Cls0']
[b'']
[b'VDID0|Prm|Mth|Bk|Foreach|VDE|Cls0']
[b'IntEx1|VD|VDE|Ex|Bk|Foreach|Fld1|Nm2']
[b'IntEx1|VD|VDE|Ex|Bk|Foreach|Fld1|This0']
[b'VDID0|VD|VDE|Foreach|Fld1|This0']
[b'Nm1|Mth|Bk|Foreach|Fld1|Nm2']
[b'']
[b'']
[b'']
[b'This0|Fld|Nm2']
[b'']
[b'']
[b'']
[b'']
[b'Nm0|Cal|If|Bk|Ret|Nm0']
[b'Cls1|Prm|Mth|Bk|Foreach|Bk|If|Cal0|Nm0']
[b'']
[b'']
[b'Nm1|Mth|Bk|Foreach|VDE|VD|VDID0']
[b'']
[b'Cls0|VDE|Foreach|Fld1|Nm2']
[b'']
[b'']
[b'']
[b'Nm3|Cal|If|Bk|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'']
[b'']
[b'Nm3|Cal|If|Bk|Ex|PosInc|Nm0']
[b'Cls1|Prm|Mth|Bk|Foreach|VDE|VD|VDID0']
[b'VDID0|VD|IntEx1']
[b'VDID0|VD|VDE|Ex|Bk|Foreach|Fld1|This0']
[b'']
[b'Prim0|Mth|Prm|Cls1']
[b'VDID0|Prm|Mth|Bk|Foreach|Fld1|Nm2']
[b'']
[b'Cls0|VDE|Foreach|Bk|If|Cal0|Nm0']
[b'']
[b'Nm1|Mth|Bk|Foreach|Bk|If|Cal0|Nm0']
[b'Cls0|VDE|Foreach|Bk|If|Bk|Ret|Nm0']
[b'Nm0|Cal|If|Bk|Foreach|Bk|Ret|Neg|IntEx0']
[b'']
[b'Cls1|Prm|Mth|Bk|Ret|Neg|IntEx0']
[b'This0|Fld|Foreach|Bk|If|Cal0|Nm0']
[b'']
[b'']]]

result 2:
Predicted: ['index', 'of', 'target']
Attention:
TIMESTEP: 0 : index
0.106182 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.094745 context: int,(PrimitiveType0)^(MethodDeclaration)(NameExpr1),METHOD_NAME
0.071685 context: object,(ClassOrInterfaceType1)^(Parameter)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.062928 context: int,(PrimitiveType0)^(MethodDeclaration)(Parameter)(VariableDeclaratorId0),target
0.045155 context: i,(NameExpr0)^(UnaryExpr:posIncrement)^(ExpressionStmt)^(BlockStmt)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.044015 context: equals,(NameExpr3)^(MethodCallExpr)^(IfStmt)^(BlockStmt)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.042547 context: METHOD_NAME,(NameExpr1)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.039831 context: int,(PrimitiveType0)^(VariableDeclarationExpr)^(ExpressionStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.036678 context: target,(NameExpr2)^(MethodCallExpr)^(IfStmt)^(BlockStmt)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.036100 context: this,(ThisExpr0)^(FieldAccessExpr)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
Attention:
TIMESTEP: 1 : of
0.068563 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.054894 context: object,(ClassOrInterfaceType1)^(Parameter)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.039301 context: int,(PrimitiveType0)^(MethodDeclaration)(NameExpr1),METHOD_NAME
0.032482 context: object,(ClassOrInterfaceType1)^(Parameter)^(MethodDeclaration)(BlockStmt)(ForeachStmt)(FieldAccessExpr1)(ThisExpr0),this
0.031575 context: i,(NameExpr0)^(UnaryExpr:posIncrement)^(ExpressionStmt)^(BlockStmt)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.030011 context: METHOD_NAME,(NameExpr1)^(MethodDeclaration)(BlockStmt)(ForeachStmt)(BlockStmt)(IfStmt)(MethodCallExpr0)(NameExpr3),equals
0.027228 context: elem,(NameExpr0)^(MethodCallExpr)^(IfStmt)^(BlockStmt)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.025339 context: elem,(VariableDeclaratorId0)^(VariableDeclarator)^(VariableDeclarationExpr)^(ForeachStmt)^(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
0.024614 context: this,(ThisExpr0)^(FieldAccessExpr)^(ForeachStmt)(BlockStmt)(IfStmt)(MethodCallExpr0)(NameExpr3),equals
0.024533 context: METHOD_NAME,(NameExpr1)^(MethodDeclaration)(BlockStmt)(ReturnStmt)(UnaryExpr:negative)(IntegerLiteralExpr0),1
Attention:
TIMESTEP: 2 : target
0.142251 context: METHOD_NAME,(NameExpr1)^(MethodDeclaration)(Parameter)(VariableDeclaratorId0),target
0.066175 context: target,(VariableDeclaratorId0)^(Parameter)(ClassOrInterfaceType1),object
0.053913 context: int,(PrimitiveType0)^(MethodDeclaration)(Parameter)(VariableDeclaratorId0),target
0.049049 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ForeachStmt)(FieldAccessExpr1)(ThisExpr0),this
0.041023 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ForeachStmt)(BlockStmt)(IfStmt)(MethodCallExpr0)(NameExpr0),elem
0.033182 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ExpressionStmt)(VariableDeclarationExpr)(PrimitiveType0),int
0.030040 context: METHOD_NAME,(NameExpr1)^(MethodDeclaration)(Parameter)(ClassOrInterfaceType1),object
0.029911 context: target,(VariableDeclaratorId0)^(Parameter)^(MethodDeclaration)(BlockStmt)(ForeachStmt)(BlockStmt)(IfStmt)(MethodCallExpr0)(NameExpr2),target
0.023267 context: elem,(NameExpr0)^(MethodCallExpr)(NameExpr2),target
0.022454 context: int,(PrimitiveType0)^(MethodDeclaration)(Parameter)(ClassOrInterfaceType1),object

[urialon]

Hi @lishi0927,
I'm not sure what are you trying to say.
You changed the reader code, therefore test/predict is not deterministic anymore.
What are you trying to do? what did you expect to happen?

[lishi0927]

En.
I want to know whether the embedded vector that is encoded by "compute-context" function is static for the same java code or not after training process.
The input of your neural work is a series of tuples, which contain (source, path, target), and you can change the order of these tuples through Random_Contexts = True. Different order usually gives different inputs for this neural network, even for the same java program.
However, different orders should return the same prediction because they are from the same java program, and your model has achieved this from the above experiments.
Also I find the embedded vector is different for different orders, but they give the same prediction.
I expect to know more about the embedded vector, because in my application, the training loss has a vibration due to different orders.
I know I use a few training examples, but usually the training loss will quickly remain stable or overfit in the small dataset.
Have you ever get my point or given me some advice?
Thank you.

[urialon]

I prefer to say that the result of the compute_context function is "deterministic", I am not sure what is "static" regarding a computation.

• After training, this resulting vector is deterministic. For the same example, the same vector will be produced.
• The order of the (source,path,target) tuples indeed doesn't matter, as you said.
• However, random_contexts doesn't only change the order, it changes the choice of the subset of all available tuples. I.e, in every training iteration, a different subset of tuples is chosen for the same example.
• Nonetheless, the random_contexts option is irrelevant for testing, it is only relevant for training. At test time (unless you change the code) - it behaves as it is set to False. I.e., at test time, always the first same max_contexts are taken (without randomness).
• If you believe that random_contexts makes your training loss fluctuate, you can simply turn it off and forget about this feature. Its effect is only 2-3 F1 points in our experiments (see Section 5 in the paper).

I hope it helps...

[lishi0927]

Thank you for your reply.
It helps me understand your paper and this network.

[lishi0927]

Another question: How many training examples in the training dataset are enough?

[urialon]

The more the merrier :-)

…

On Wed, Dec 11, 2019 at 10:41 AM Paul Lee ***@***.***> wrote: Another question: How many training examples in the training dataset are enough? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#26?email_source=notifications&email_token=ADSOXMBVXWEL5NC5SH67E6DQYCRTJA5CNFSM4JTCMB32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGSKEQQ#issuecomment-564437570>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADSOXME7VUWZNGCGJ62HHB3QYCRTJANCNFSM4JTCMB3Q> .

[lishi0927]

But how to validate the generalization ability of this neural network?

[urialon]

The only way I know is by testing it on a distinct validation set...

…

On Wed, Dec 11, 2019 at 2:56 PM Paul Lee ***@***.***> wrote: But how to validate the generalization ability of this neural network? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#26?email_source=notifications&email_token=ADSOXMGSPGV5CU3YHB2DIZ3QYDPQXA5CNFSM4JTCMB32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGTADNI#issuecomment-564527541>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADSOXMBQGVPPWVLHSNBDPU3QYDPQXANCNFSM4JTCMB3Q> .