kefirski/pytorch_neg_loss

NEG loss implemented in pytorch

pytorch_neg_loss's Issues

How is sum_log_sampled implemented?

According to the original word2vec paper (https://arxiv.org/pdf/1310.4546.pdf), Negative sampling loss is implemented as:

Your implementation for the sum log sample is:

sum_log_sampled = t.bmm(noise, input.unsqueeze(2)).sigmoid().log().sum(1).squeeze()

It's the same as described in the paper except the minus sign, isn't it look like below

sum_log_sampled = (-1* t.bmm(noise, input.unsqueeze(2))).sigmoid().log().sum(1).squeeze()

Am I right ?
https://github.com/kefirski/pytorch_NEG_loss/blob/master/NEG_loss/neg.py#L75

Why using two embedding layers?

I noticed that two embedding layers are used in your code, 'out_embed' and 'in_embed', for output and input respectively, and only 'in_embed' weights are saved.

But can they just share one embedding layer since they are in the same corpus？

how does the negative sampling work?

Hi, sorry for a newbie question, how does the negative sampling work in neg.py, in particular line 69? I cannot find any documentations on calling neg in nn.embedding...

Subsampling

Hi.
Why not use torch.multinomial instead of np.random.choice?