Comments (4)
read through the paper, I didn’t find what w2v embedding other models(such as LSTM,CNN) are using. It is amazing that SWEM -ave can achieve better results than LSTM or CNN in some tasks, which in fact I don’t believe! I have done a lot of nlp tasks and I know that simply average the word embedding of a text is usually very poor.
I don’t think the comparisons of other models are fair. They don’t even use the same pretrained w2v. So maybe it’s just the Glove you used is better than the embedding other models used.
from swem.
To your questions/concerns:
First, the same word embeddings (Glove) were used for both LSTM and CNN to ensure fair comparison.
Second, you should be really careful while implementing the simple average operation to obtain sentence embeddings. Specifically, a mask is needed to remove the influence of padding tokens. Otherwise the paddings will hurt the final performance a lot. This is what I have implemented in this codebase.
Finally, we have tried to shuffle the words in every sentence and the LSTM gives the same accuracy. In other words, LSTM does not leverage the word order information at all, or the information is not useful for the tasks considered.
from swem.
Thanks for replying so fast.
In paper [Character-level Convolutional Networks for TextClassification] where the results of LSTM/bag-of-means/CNNs are reported, I couldn't find the evidence that they were using Glove. Actually, that paper didn't even cite Pennington's Glove paper. I only noticed that both of you used the same dimension of w2v---300. So I still have doubts for this.
As for the trick for average pooling that mask should be applied, I also noticed it from your code. I will try it and see its effectiveness. Thanks for this important information.
Although I still have soncerns,I believe all the results in your paper are true. So if the simple baseline can beat so many powerful complicated models such as LSTM and CNNs, dose that mean the LSTM and CNNs and fasttext have no use, which is widely implemented in industry and reasearch? Because even they are slightly better than SWEM, their computational cost is much higher.
from swem.
Is the LSTM or CNN model implemented with mask?
from swem.
Related Issues (14)
- aver_emb_encoder HOT 1
- Not find SWEM-hier HOT 16
- requirement missing gensim matplotlib
- how can SWEM-max train a wordEmbedding
- where can i get the Yelp P. and Yelp F. dataset?
- Difference between the paper and the implementation HOT 1
- About NASH
- Why simple average pooling of w2v can
- Package dependency related errors occuring while loading snli_emb.py HOT 1
- executing requirement.txt leaves you with a CPU backed tensorflow
- Experiment Setting on Section 4.1.1 in the paper
- Floating hitching post and riding horses freezes and crashes the game.
- 1.18.2 I used the Mob Capture Device by tfarecnim to get my horse out of the water. Now its model is frozen. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from swem.