rikeda71/char-blstm-crf-for-japanese

An Inplementation of Character based Bidirectional LSTM CRF for Japanese

char-blstm-crf-for-japanese's Issues

pickle error when running the predict_sample.py code

code:

from deepjapaner.modelapi import ModelAPI

api = ModelAPI(model_path=r'D:\CRF_Jap\model_1_jap.pth', train_path=r'D:\japanese_ner_datasets\IOB2Corpus-master\hironsan.txt',
               wordemb_path='D:\CRF_Jap\ja_tar\ja.txt', charemb_path=r'D:\CRF_Jap\ja_tar\ja.txt',
               hidden_size=300)

label = api.predict('私は六花亭の白い恋人を食べました')
print(label)

ERROR looks like:

Traceback (most recent call last):
  File "D:/CRF_Jap/Char-BLSTM-CRF-for-Japanese-master/predict_sample.py", line 5, in <module>
    hidden_size=300)
  File "D:\CRF_Jap\Char-BLSTM-CRF-for-Japanese-master\deepjapaner\modelapi.py", line 35, in __init__
    self.model.load(model_path)
  File "D:\CRF_Jap\Char-BLSTM-CRF-for-Japanese-master\deepjapaner\model.py", line 62, in load
    self.load_state_dict(torch.load(model_path))
  File "C:\Users\kavya\Anaconda3\lib\site-packages\torch\serialization.py", line 367, in load
    return _load(f, map_location, pickle_module)
  File "C:\Users\kavya\Anaconda3\lib\site-packages\torch\serialization.py", line 528, in _load
    magic_number = pickle_module.load(f)
_pickle.UnpicklingError: pickle data was truncated

Process finished with exit code 1

I had created a .pth file and added my site-packages path in that file ( C:\Users\kavya\Anaconda3\Lib\site-packages )

What can be the reason?

Should I make any other changes in the codes. Can you please help me with this?

training data

what are the wordvectors and charvectors

Do we need to use IOB2 corpus where both word and character embeddings are given
Can you please help through this?

Implement early stopping

AttributeError: 'Example' object has no attribute 'label'

File "/mnt/raid1//****/Char-BLSTM-CRF-for-Japanese/deepjapaner/dataset.py", line 44, in init
self.LABEL.build_vocab(self.dataset)
File "/home//anaconda3/envs/BLSTM/lib/python3.5/site-packages/torchtext/data/field.py", line 298, in build_vocab
for i,x in enumerate(data):
File "/home/*****/anaconda3/envs/BLSTM/lib/python3.5/site-packages/torchtext/data/dataset.py", line 163, in getattr
yield getattr(x, attr)
AttributeError: 'Example' object has no attribute 'label'

I meet this error . How can I fix it ..?

学習後，結果で表示する評価した固有表現数が違うバグがある

推定性能の評価

推定性能（precision, recall, F1）を出せるようなメソッドを用意する

figure1 cnn kernel size

thanks for sharing! in figure 1 of paper,how many filters in cnn and each filter size is how