BatchNorm is not in 'test mode' inside update_train_data

[martin-gorner]

BatchNorm is not in 'test mode' inside update_train_data

[martin-gorner]

Good question. It will not change anything in the training. The values computed here are for visualisation only. So question is: will this change produce "more relevant" curves on the graphs ?

If this was dropout, then yes, definitely, you need to disable dropout when you compute the test and training curves, otherwise your training curve, computed on a half-brain-dead (dropped out) network will be very bad, sometimes worse than the test curve which should never happen (test loss is normally always higher(=worse) than training loss).

For batch norm, maybe that is the case as well. You have to try. I implemented this in this way because of the way batchnorm works. It uses statistics computed on the current batch of training data. When measuring performance on your test dataset, there is no "current batch of training data" so you have to use some other stats (running average of multiple previous batches). That is what the tst:True flag enables. When measuring your training metrics, the current batch exists and using stats computed on it for batch norm is not a problem so I did not feel I should enable a solution for a problem that I did not have. But maybe I was wrong there.

[DraTeots]

Thank you very much for the explanation