Comments (21)
@ChenMiya ๆๆณ้ฎไธไธไฝ ็ฐๅจๆพๅฐไฝ ๆต่ฏ้็ๅ็กฎ็็นๅซไฝ็ๅๅ ไบๅ๏ผ
from acnn.
The question is how to calculate the score if there're several words in an entity.
@GatsbyUSTC In some papers, mean embeddings of those words are often used, too.
some reference
Recurrent Attention Network on Memory for Aspect Sentiment Analysis
Aspect Level Sentiment Classification with Deep Memory Network
Effective LSTMs for Target-Dependent Sentiment Classification
from acnn.
@GatsbyUSTC I agree with you. If we are using pretrained embeddings and not tuning them during training, an attention at the input layer would be meaningless. However, if embeddings are randomly initialized and updated while training, the attention would be useful to choose words which are key indicators for a class. At the same time, this would require a considerably larger training set.
from acnn.
@wangtianyiftd Thanks very much. Could I ask you another question since you seem to read lots of papers about attention. The attention mechanism works well on translation model. A word can focus on several words from another language based on attention. But do you think the attention mechanism will work on relation classification problem? Inner product means euclidean distance between embeddings. when we do relation classification, I don't think more attention on words which are close to entities will help solving the relation classification. I didn't get a better result after adding attention mechanism on cnn model, and I don't know the problem comes from the attention or hyperparameter. It will be nice if you can answer this question.
from acnn.
@GatsbyUSTC Attention on words try to find the most relevant words related to the entity pair, of course it will help. Take the relation "President" as an example, the word 'president' itself contribute a lot to the classification results. In theory, attention mechanism will assign a large weight to the word 'president'.
Although we can't reproduce the results of this paper, we can't deny the entire attention mechanism. Maybe you can reproduce other attention based paper.
from acnn.
I can't reproduce the results of the paper, maybe the code is not right. I don't know what's wrong.
from acnn.
@FrankWork my version which has achieved acnn is using pytorch, some data handling code copy from yours. maybe you could see it
from acnn.
@FrankWork thanks for your recommendation
from acnn.
@lawlietAi What is your test accuracy?
from acnn.
that is the reason i didn't put it on README
from acnn.
@FrankWork i think the original paper didn't use test data but use the eval data which divided from train data
from acnn.
Is there anyone actually obtaining the results from the original paper? I couldn't reach the performance specified in the paper.
from acnn.
@allanj no one.
from acnn.
@lawlietAi I'm not familiar with pytorch while I'm implementing the paper using tensorflow. And I'm stuck on a specific problem. In original paper, the author views an entity as a word and calculate a similarity score between them. But in fact, there're entities composed of several words. How do you deal with that? how to calculate the score in such situation?
from acnn.
sorry, i could not understand certainly on โthemโ. ๐๐
from acnn.
@lawlietAi Sorry. In input attention layer, the author calculate a score f between word in entity and word in sentence based on inner product. The question is how to calculate the score if there're several words in an entity.
from acnn.
@GatsbyUSTC If an entity contains several words, we just use the last word to represent the entity. It is a common practice.
from acnn.
attention is not almighty. but it can let network get the keywords from articles. this paper used 2 attention layers to increase the depth of nn, but att reduced information a lot. maybe it is the reason of bad performance
in addition, maybe this paper gave us a way to process long articles. i would do more experiment further
from acnn.
@FrankWork I did not deny the entire attention mechanism. The question is how can the attention learn to find the most relevant words on relation classification task. The attention mechanism in input layer proposed by the author just compute a inner product between entity and words so that words that are similar to entity will be paid more attention. Does it really help relation classification? Take "The school master teaches the lesson with a stick" for example, keyword for relation classification should be "with" since the correct relation is Instrument-Agency. Word "with" is relevant to the relation classification but it doesn't seem to be similar with "master" or "stick" which means we can't get a high attention score based on inner product. I just want to know the function of the attention mechanism here. Of course I know in some applications the attention mechanism will help, but they also get a good reason.
from acnn.
@GatsbyUSTC The inner product does not guarantee that the two words must be similar semantically. It can also be interpreted as relevance since the embeddings are learned. "with + n." is common in English, why can't they have higher inner product?
from acnn.
@FrankWork i think the original paper didn't use test data but use the eval data which divided from train data
You codes don't have the train or test data,So how to run your codes?
from acnn.
Related Issues (8)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from acnn.