Comments (13)
from neuronlp2.
I use the database UKPLabhttps://github.com/UKPLab/acl2017-neural_end2end_am/tree/master/data/conll/Paragraph_Level run NERCRF.py and bi_lstm_cnn_crf.py they display diferent result.
NERCRF.py run result precision recall F1 almost zero:
Epoch 1 (LSTM(std), learning rate=0.0100, decay rate=0.0500 (schedule=1)):
train: 96 loss: 1050766784.1735, time: 351.21s
dev acc: 11.67%, precision: 0.00%, recall: 0.00%, F1: 0.00%
best dev acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
best test acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
Epoch 2 (LSTM(std), learning rate=0.0095, decay rate=0.0500 (schedule=1)):
train: 96 loss: 102558966.3255, time: 258.39s
dev acc: 11.69%, precision: 0.00%, recall: 0.00%, F1: 0.00%
best dev acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
best test acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
Epoch 3 (LSTM(std), learning rate=0.0091, decay rate=0.0500 (schedule=1)):
train: 96 loss: 47132442.5896, time: 257.53s
dev acc: 11.67%, precision: 0.00%, recall: 0.00%, F1: 0.00%
best dev acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
best test acc: 0.00%, precision: 0.00%, recall: 0.00%, F1: 0.00% (epoch: 0)
......
from neuronlp2.
I use the eval to count f1 :
@ManniSingh @XuezheMax
processed 12227 tokens with 501 phrases; found: 6261 phrases; correct: 0.
accuracy: 16.33%; precision: 0.00%; recall: 0.00%; FB1: 0.00
Claim: precision: 0.00%; recall: 0.00%; FB1: 0.00 2
MajorClaim: precision: 0.00%; recall: 0.00%; FB1: 0.00 0
Premise: precision: 0.00%; recall: 0.00%; FB1: 0.00 6259
why they are zero?
one of tmp/942fb2_dev11
942fb2_dev11.txt
from neuronlp2.
It seems PyTorch problem, you should clean restart.
from neuronlp2.
from neuronlp2.
Yes ,I have the tmp dir ,the dir contain dev predicton file and score file.
I just confuse why run NERCRF.py and bi_lstm_cnn_crf.py they display diferent result.
use database https://github.com/UKPLab/acl2017-neural_end2end_am/tree/master/data/conll/Paragraph_Level
from neuronlp2.
hello, Max, thank you for your great contribution on the awesome work.
I met the exactly same problem because I use the same dataset as described by @chzeze .
Does anyone solved this problem?
If there exist any solution, please let me know.
**thank you all of you **
from neuronlp2.
Hi @hey-bruce and @chzeze ,
I guess I have found the reason. Each line in the data you provided are separated by '\t', where the format in the NERCRF.py is whitespace ' '.
from neuronlp2.
thank you for your reply.
I reformated the data. and now each line in the date are separated by ' '(whitespace).
the problem is also there.
I can offer you the data through my github repo, can you test it?
thank u
from neuronlp2.
Yes, please share me the data.
Thanks.
from neuronlp2.
You can get the data from here
thank you.
the motivation that I used your method is I am trying my best to reproduce the artical. And the author used your repo "LasagneNLP"。thank you again
from neuronlp2.
Hi @hey-bruce ,
My previous reader cannot handle multiple continuous blank lines. I have revised my code to handle it.
Now the stats info can match the one reported in the paper.
But the performance is still zero. I guess it is not an issue for the model. Please first check if you use the model the right way. Second, please make sure that the evaluation script for NER is suitable for the new task. The evaluation script used in my code is from CoNLL 2003 shared task, which is designed for NER.
Before you run your code locally, please make sure that you do the following two things:
- git pull to get the latest version.
- remove the data/alphabets/ folder to create a new one. If the code detect the folder, it will assumes that the alphabets have already been created and will try to load them from disk.
from neuronlp2.
I really appreciate your help.
I tried it just now. And yes, it's not your model's issue and the evaluation script is not suitable for my task. maybe I should find a new strategy to evaluate it.
And I think you can close this issue now
from neuronlp2.
Related Issues (20)
- RuntimeError: maximum recursion depth exceeded (python3+torch0.4) HOT 4
- TypeError: invalid file: None ===conllx_stacked_data.create_alphabets(alphabet_path, None,
- No such file or directory: 'data/sskip/sskip.ger.64.gz' && data/sskip/sskip.eng.100.gz && data/conll2003/english/eng.train.bioes.conll HOT 3
- whart is /run_analyze.sh used for, looks missing some files HOT 1
- Time to Compute
- Run a trained model HOT 1
- RuntimeError: maximum recursion depth exceeded HOT 2
- some questions about dataset and f1 score! HOT 1
- Can you give me the data and sskip.eng.100.gz? HOT 2
- Error Analysis
- how to use Elmo or Bert HOT 1
- embedding size HOT 1
- parsing input format HOT 3
- variability in the results HOT 2
- RuntimeError: CUDA error: an illegal instruction was encountered HOT 3
- Interpretation of code variables HOT 2
- Error trying to train a model HOT 2
- AssertionError for word_dim HOT 2
- Unable to find training data HOT 2
- TypeError in Biaffine HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from neuronlp2.