π¬ Β AI Researcher
π Β B.S. in Math, CS @ Yonsei Univ. π¦
βοΈ Β Ph.D in CSE @ SNU πͺ
π Β CV
Skipgram Negative Sampling implemented in PyTorch
License: MIT License
π¬ Β AI Researcher
π Β B.S. in Math, CS @ Yonsei Univ. π¦
βοΈ Β Ph.D in CSE @ SNU πͺ
π Β CV
torch
tqdm
Is the stand for the word unknown?
As I can see the 'ws' variable representing discard probabilities of each word is unused. Should it be applied for calculating weights?
In this formula we have an expectation of
In your code, you minimized -(oloss + nloss).mean()
which means (oloss+nloss) should be large.
So, "oloss become large and nloss become small " is expected.
Although -(oloss+nloss) decrease, I got oloss become small and nloss become large, how so?
First, thanks for you excellent code :)
In model.py, the following piece of code suggests that we may get positive word when we do negative sampling, though the probability is very small.
nwords = t.multinomial(self.weights, batch_size * context_size * self.n_negs, replacement=True).view(batch_size, -1)
I'm wondering why you didn't perform equality check, is that because it doesn't affect the quality of trained word vectors but slow down the training speed?
Are there other reasons?
I download your code and trained among 100 million sentences, eventually I got a loss of 0.44, can it be a good one?
Why code below, in the project, can be used as "loss".
oloss = t.bmm(ovectors, ivectors).squeeze().sigmoid().log().mean(1) nloss = t.bmm(nvectors, ivectors).squeeze().sigmoid().log().view(-1, context_size, self.n_negs).sum(2).mean(1)
In my judgment, "loss" should be "prediction" - "actual result".
But, in the upper code, "oloss" is prediction, without operation on actual result.
Hi,
Can you please explain, what is the purpose of including the <UNK>
tokens in the owords
vector produced by skipgram
function? What should model learn by using these as training examples?
Also, what is the purpose of variable ws
in train
function, if it's not used anywhere after its definition?
Hey there, great skipgram example, so thank you for that.
I have a question on why you decided to use different embeddings for the "input" words and "output"/"negative" words? See lines below:
https://github.com/theeluwin/pytorch-sgns/blob/master/model.py#L29:L30
I imagine this could give better performance on some problem, but haven't been able to test this myself yet. Thanks for the help!
Hi theeluwin!
First of all thanks for the code, it was well written and helped me a ton in building my own word2vec model.
This is not an issue per se, but something I'm potentially adding to the word2vec model using your code, the main idea is to use regularisation on embeddings in a temporal setting. I've run into trouble with the code and I'm wondering if you'd be so kind as to help out!
the main idea is that I'm training 2 sets of models (model 0 & 1) consecutively based on 2 sets of corpora, the 2 sets are temporally adjacent (say news articles of 01/jan and 02/jan), during the training of model 1, I'd like to add a penalty term to the loss/cost function:
for all the words in set(vocab_0)&set(vocab_1), I'd like to minimise the distance of the same word's embeddings from period 0 & 1.
I'm not sure if it makes sense!
So far I'm testing on embeddings of rather small dimensions ~ 20, therefore I'm using the Euclidean distance as a measure.
based on your code, I added a fordward_r function in the Word2Vec class:
`
def forward_r(self, data):
if data is not None:
v = LT(data)
v = v.cuda() if self.ivectors.weight.is_cuda else v
return(self.ivectors(v))
else:
return(None)
`
This function simply extracts the relevant embeddings (words from the intersection of the 2 vocabs)
and then in the SGNS, I'm now only testing on 1 particular embedding, I added the following loss calculation that look like this:
rvectors = self.embedding.forward_r(rwords)
rloss = 3*((rvectors.squeeze() - self.vector3)**2).sum()
and finally it woud return the following total loss:
return -(oloss + nloss).mean() + rloss
However the problem is, the loss gets stuck, it never updates, and it appears that the back propagation is not working properly.
As you can probably tell, I'm rather new to pytorch and I'm not sure if you could lend me a hand on what's happening!
Thank you so much in advance!
Hi
Thank you for sharing the code. However, when I tried to test the repo with "python preprocess.py" and " python train.py --weights --cuda", the first one worked well and generated processed data, whereas the second reported the error as follows:
[Epoch 1]: 0%| | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
File "train.py", line 93, in train(parse_args())
File "train.py", line 81, in train loss = sgns(iword, owords)
File "/home/weixin/anaconda2/envs/p3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in call
result = self.forward(*input, **kwargs)
File "/home/weixin/Downloads/pytorch-sgns-master/model.py", line 70, in forward
ivectors = self.embedding.forward_i(iword).unsqueeze(2)
File "/home/weixin/Downloads/pytorch-sgns-master/model.py", line 42, in forward_i
return self.ivectors(v)
File "/home/weixin/anaconda2/envs/p3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 325, in call
result = self.forward(*input, **kwargs)
File "/home/weixin/anaconda2/envs/p3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 103, in forward
self.scale_grad_by_freq, self.sparse
RuntimeError: save_for_backward can only save input or output tensors, but argument 0 doesn't satisfy this condition
I am quite new to Pytorch so any idea what might go wrong?
Many thanks.
The loss function currently implement is
-(oloss + nloss).mean()
It should be
(-oloss + nloss).mean()
You want to minimize the distance between "positive samples" and maximize the distance between "negative samples".
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.