Questions about NCEAverage.py

[hobbitlong]

Questions about NCEAverage.py

[vinsis]

Update: I realized torch.no_grad() is to prevent the issue of buffers being cleared.

In reference to the last point, after some thought I think I realize why torch.no_grad() is used. It is to make sure the gradients associated with l and ab are not stored in the memory bank.

[vinsis]

In reference to the third point, I think one need not define a context as was done in CPC. A simple dot product followed by likelihood maximization should be enough. There is no parameter to be optimized.

[vinsis]

I think I figured out answers to most of the questions I asked. However I still don't know the purpose of T. I would really appreciate it if you could answer. Thanks.

[HobbitLong]

Hi, sorry for missing the message.

So T is used to adjust the dynamic range of the score before softmax-ce. As the features are normalized, inner product only gives you something between -1 and 1, which is insufficient for discriminative positives from negatives. So T plays the role to adjust the range.

[vinsis]

Ah I see. So T is a hyperparameter whose value was decided empirically I guess. Thank you!
Closing the issue now.