Comments (5)
For second case, in our original experiments, we used tuple, i.e. (event type, role), to do evaluation, and classify N/A class only and if only event type is N/A or role is N/A. This code is fully rewritten because the origin code is quite dirty and I apologized that I missed this point in this rewritten code. Now I fixed this point , please check commit history to find out the modification, I also tested the modified codes and the results remained same as the paper.
Just a reminder, once you use tuple to do classification, since the evaluation is more strict, you need to do some more process in dataset to reach 53.5 in DMCNN, such as cutting sentences if its length is more than a threshold, please do such process by yourself. Also, from the experiments results, once DMCNN reaches 53.5 using these two evaluation metrics separately, then other models will perform similarly using these two evaluation metrics separately, which means it is just a trick in data preprocess, and this is the one of main disadvantages of ACE2005.
For first case, when a trigger is classified with N/A, it is not necessarily to do EAE stage with this trigger because all entities are N/A in such situation and are totally noises when testing EAE. Also, actually all models are tested in this way, if you want to see what will happen if you take the first case into consideration, you can simply comment line 167,168 in models.py
,i.e.
dev = [np.take(d,dev_slices,axis=0) for d in dev]
test = [np.take(d,test_slices,axis=0) for d in test]
You will see that there will be a dramatically drop in performance and DMCNN will have no way to reach 53.5 since there are too many meaningless noises. From this perspective, hope you can understand what happened here.
from hmeae.
Thank you for your response.
I'm okay with the second case and thank you for modification.
For the first case, if we mislabel a trigger as None type, the trigger and its arguments are not included in EAE stage for performance calculation. I got your point but it doesn't looks like a right way.
Thank you again for modification.
from hmeae.
Another question is what event detection model is used when you report HMEAE(CNN) and HMEAE(BERT) in Table 3/4? Is DMCNN used as ED model? The experimental setting is confusing for me since you used two event detection models:
As our work does not involve the event detection stage, we conduct the ar- gument role classification based on the event detection models in Chen et al. (2015) and Wang et al. (2019) for HMEAE (CNN) and HMEAE (BERT) respectively.
Or HMEAE(CNN) used DMCNN as ED model and HMEAE(BERT) used model from Wang et al. (2019) as ED model?
Thank you.
from hmeae.
Second one, i.e. DMCNN as ED + HMEAE(CNN) as EAE, DMBERT as ED + HMEAE(BERT) as EAE.
from hmeae.
Got it and Thanks!
from hmeae.
Related Issues (13)
- question about data format HOT 6
- Confusion about the output (role type) HOT 4
- Question about performance HOT 3
- Questions about f_score HOT 2
- Could you release HMEAE(BERT) model ? HOT 2
- Question about the special token which indicates event type. Thank you HOT 1
- Question about TAC KBP HOT 2
- Question about extracting information of ACE 2005
- json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) HOT 3
- Tips: only support stanford-corenlp-full-2018-10-05 HOT 1
- stanford-english-corenlp-2018-10-05-models.jar HOT 2
- TypeError: word_tokenize() takes 2 positional arguments but 3 were given HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hmeae.