bollu / alok-bollu Goto Github PK
View Code? Open in Web Editor NEWA repo for work between Alok Debnath and Siddharth Bhat
A repo for work between Alok Debnath and Siddharth Bhat
10
A[i] = i
A[i][j]
with i*j
.atomicAdd(&result[i], a[i] + b[i])
w^tX
. Parallelise on mini-batches?word2vec
Running on branch alok-evaluation, as branch master does not seem to have a models directory.
Command: python3 run.py eval models/working-word2vec-baseline-b0c2872c
Output:
setting up device...
device: cuda:1
====time: 0:00:00.616033
loading model from: models/working-word2vec-baseline-b0c2872c...
Traceback (most recent call last):
File "run.py", line 758, in <module>
PARAMS = torch.load(PARSED.loadpath, map_location=DEVICE)
File "/home/alokdebnath5/.local/lib/python3.5/site-packages/torch/serialization.py", line 367, in load
return _load(f, map_location, pickle_module)
File "/home/alokdebnath5/.local/lib/python3.5/site-packages/torch/serialization.py", line 538, in _load
result = unpickler.load()
AttributeError: Can't get attribute 'Word2Man' on <module '__main__' from 'run.py'>
`
Need help in figuring out why this is happening?
Plot vector length against word frequency. Since the word2vec-sn-reimann
did not do well, this seems to indicate that training for dot-product 1 does worse than the regular word2vec, for whatever reason. Understand why
We need to look into the SoTA word embeddings evaluation testbench, in order to analyze the our performance for vectors in downstream tasks
v.w
, not sigm(v.w)
cos(theta)
, not sigm(v.w)
let vec(w) . vec(w') = 1.0 - delta(neighbor(w), neighbor(w'))
. If the neighbourhood structure of w
and w'
are the same, then we should make them dot product 1
, otherwise we should make them dot product based on how close their neighborhoods are.
The kind of optimisation that glove tries to perform (finding vectors whose dot products are known) is a well known type of program as far as I understand: In particular, it forms a semidefinite program (SDP)
Here are the possible scenarios:
In this case, we use a solver for SDP against their corpus of co-occurences.
Publish, show why NNs + gradient descent are unnecessary for solving SDPs.
There are two reasons for this. Either:
Their gradient descent mechanism finds vectors whose dot products better match the co-occurence matrices than a solver for SDP. In this case, we publish in the SDP literature about how gradient descent is better than a fucking solver (highly unlikely) $$$
our model has dot products that fits the co-occurence ratios better than glove, but still performs worse ( $$$)
This opens up a line of investigation, since it shows that their "explanation" is BS. Because our model literally optimizes what they claim they are optimizing better, and yet performs worse on the results. This allows us to argue one of:
:((
We try to massage it into an SDP, but this is the most disappointing possibility
word-analogy.cpp
to use the new mulQuadForm
code. The old code is deprecated and should be removed from word-analogy.cpp
. To test that this is correct, change the functionvoid setupDotContainmentMat(int n, real *m) {
const int d = log2(n);
assert (1 << d == n);
for(int i = 0; i < n; ++i)
for(int j = 0; j < n; ++j)
m[i*n+j] = i == j ? 1 : 0;
}
we recover word2vec
.
Hey,
Please implement the fill in the blank based evaluation method: I ate a ___ on the bus: Guess the ___ based on the average of the vectors around it.
Figure out what downstrem evaluation task is to be used to validate our models
Need to identify and consider the changes in the code of word2vec to store syn1neg as well, and also to load syn1neg while doing distance and word analogy in order to capture the true meaning of contextualized similarity.
Issue 1: Eval versus wordnet eval in the parameters are interchangably used
Issue 2:
setting up device...
device: cuda:3
====time: 0:00:00.983759
loading corpus: text8...
Traceback (most recent call last):
File "run.py", line 1123, in <module>
main()
File "run.py", line 1076, in main
PARAMS = Parameters.load_model_state_dict(LOGGER,
NameError: name 'Parameters' is not defined
Switch out the dot product of word2vec-ga
to be like word2vec
. Make a commit with this change, and train the model. Check if the model reproduces word2vec
. If not, let's sit together and debug.
Basically we suspect that even tho the training occurs on 300 dimensions, the actual structure is captured by a much lower dimensional space.
In particular, we should look at conformal dimensionality reduction methods i.e. dimensionality reduction methods which preserve angles.
L1
lossL2
losshttps://arxiv.org/pdf/1412.6623.pdf
This is an ICLR 2015 paper that talks about Gaussians for a word representation. We need to use their evaluation section for out paper.
Command: python3 run.py train --savepath ./models/
Error:
setting up device...
device: cuda:3
====time: 0:00:01.179288
loading corpus: text8...
Traceback (most recent call last):
File "run.py", line 1186, in <module>
main()
File "run.py", line 1167, in main
traintype=PARSED.traintype)
File "run.py", line 661, in __init__
self.NDOCS = int(NDOCS)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType' ``
Please explain this as well. Thanks
Setup our codebsse such that we can use the gensim.word2vec
implementation as well.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.