aiforsec/ladder

ladder's Issues

Sharing the complete dataset & Fix for tokenizer

Hello,

firstly, amazing paper and great work! Thank you!

I was wondering if there is a dataset version that includes all annotations? It seems like the dataset is split into several files and there seems to be some overlap between examples. I am not sure what kind of file format BRAT returns for its annotations but a complete json file would be great!

The dataset for the 5 different malware reports would also be great to have!

The fix for the tokenizer problem in the jupyter notebook is to use python3.8 as far as I can tell.
Also the requirements.txt should use scikit-learn not sklearn. I also suggest using the latest version of sentence_transformers instead of 2.2 like in the notebook.

Annotation Errors in train.txt

There are annotation mistakes in the following lines in ner/dataset/150/train.txt

1252 
17704 
21348 
55951 
71407 
76929 
89454
89849 

A new entity also shouldn't start with "I" this happens a lot of times as well.

7578
9123
10955
21145
21146
25173
25305
25309
25315
25545
26543
44592
44593
44846
44991
45306
55892
56102
59409
59456
59589
59783
59919
60028
60039
60043
60058
71250
71406
72246
74812
74998
75000
75170
75202
75204
89389
89533
89643
89688
89760
89794
89957
101854
102514

Just an FYI