Comments (5)
Hi Andy, thanks very much for taking the time to create an issue. Looking at the code, it looks like I was unsure about this too as I left this comment:
# or when it simply does not appear in the test set, then it is
# spurious, but it is not clear where to assign it at the tag
# level. In this case, it is applied to all target_tags
# found in this example. This will mean that the sum of the
# evaluation_agg_entities will not equal evaluation.
What do you think about it @davidsbatista?
from ner-evaluation.
Also @anjiefang you may be interested to see that we started to convert this code into a module here https://github.com/ivyleavedtoadflax/nervaluate, although we've not got too much further yet. I have a task coming up for which I will need to use it, so I hope to get more time to develop it in the near future.
from ner-evaluation.
I'm working with the library and I've found a (i think) mistake in this part of the code.
When the predicted entity is not in the true-entities list and the offsets do not match exactly with any of the true-entities and it do not have any overlap with any of the true-entities the code add 1 to all the labels in the 'spurious' field. This is because the note:
NOTE: when pred.e_type is not found in tags
or when it simply does not appear in the test set, then it is
spurious, but it is not clear where to assign it at the tag
level. In this case, it is applied to all target_tags
found in this example. This will mean that the sum of the
evaluation_agg_entities will not equal evaluation
but there is no check to ensure that the predicted label is not in the label set.
Maybe it is neccesary to add if pred.e_type not in tags:
before the for true in tags:
here?
from ner-evaluation.
Hi @amlarraz. Many thanks for your comment. Could you possibly open a PR for this?
from ner-evaluation.
No problem, I've just created the pull request.
Many thanks for your work!
from ner-evaluation.
Related Issues (16)
- sklearn_crfsuite add bigram feature? HOT 1
- Create module and accept other formats HOT 8
- Different possible input formats HOT 3
- range is wrong for only 1 token span HOT 2
- find_overlap question HOT 3
- true_which_overlapped_with_pred does not get updated properly
- collect_named_entities skip over entities HOT 5
- Aggregated by entity type results possible error HOT 3
- .
- Loop breaks stop looking for predicted entities that may overlap with true entities
- Pin scikit-learn==0.23.2 HOT 1
- Incorrect entities extraction HOT 2
- Question: Are Partial Matches allowed for Type matching scheme? HOT 2
- Some problems with evaluation_agg_entities_type HOT 9
- True entities considered multiple times HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ner-evaluation.