louis-udm/ner-bert-crf

NER implementation with BERT and CRF model

Zhibin Lu

This is a named entity recognizer based on BERT Model(pytorch-pretrained-BERT) and CRF.

Someone construct model with BERT, LSTM and CRF, like this BERT-BiLSTM-CRF-NER, but in theory, the BERT mechanism has replaced the role of LSTM, so I think LSTM is redundant.

For the performance, BERT+CRF is always a little better than single BERT in my experience.

Requirements

• python 3.6
• pytorch 1.0.0
• pytorch-pretrained-bert 0.4.0

Overview

The NER_BERT_CRF.py include 2 model:

• model 1:
  • This is just a pretrained BertForTokenClassification, For a comparision with my BERT-CRF model
• model 2:
  • A pretrained BERT with CRF model.
• data set
  • CoNLL-2003

Parameters

• NER_labels = ['X', '[CLS]', '[SEP]', 'O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']
• max_seq_length = 180
• batch_size = 32
• learning_rate = 5e-5
• weight_decay = 1e-5
• learning_rate for CRF and FC: 8e-5
• weight_decay for CRF and FC: 5e-6
• total_train_epochs = 20
• bert_model_scale = 'bert-base-cased'
• do_lower_case = False

Performance

• Bert paper
  • F1-Score on valid data: 96.4 %
  • F1-Score on test data: 92.4 %
• BertForTokenClassification (epochs = 15)
  • Accuracy on valid data: 99.10 %
  • Accuracy on test data: 98.11 %
  • F1-Score on valid data: 96.18 %
  • F1-Score on test data: 92.17 %
• Bert+CRF (epochs = 16)
  • Accuracy on valid data: 99.10 %
  • Accuracy of test data: 98.14 %
  • F1-Score on valid data: 96.23 %
  • F1-Score on test data: 92.29 %

References

• Bert paper
• Bert with PyTorch implementation
• ericput/Bert-ner
• CoNLL-2003 data set
• Kyubyong/bert_ner

ner-bert-crf's Issues

Running Error

Hello,

I am trying to run the code and got this error!
can you help please

Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.
Python version 3.7.12 (default, Jan 15 2022, 18:48:18)
[GCC 7.5.0]
PyTorch version 1.10.0+cu111
Traceback (most recent call last):
File "NER_BERT_CRF.py", line 65, in
set_work_dir()
File "NER_BERT_CRF.py", line 50, in set_work_dir
raise Exception('Set work path error!')
Exception: Set work path error!

Running code ERROR

I think line 875 at function DataLoader, the collate_fn=pad should be collate_fn=NerDataset.pad ...

Are there any plans to avoid using the special token X?

I think that the special token X was used in BERT paper v1 but not used in v2.

BERT paper v2 is written below.

5.3 Feature-based Approach with BERT
 We use the representation of the first sub-token as the input to the token-level classifier over the NER label set.

In the middle of Chapter 5.3, it is written that the label is set on the first token of the subword. Since there is no description of "X", "X" is not used at present.

The following issue also states that the model can learn patterns by learning only the first word of the subword with fine-tuning without using'X'.
kamalkraj/BERT-NER#1 (comment)

Are there any plans to avoid using the special token X?

如何用训练的模型做接口展示

accuracy to 0 when CRF_only=False

Hi, I borrowed your functions, as below, for training CoNLL_2003 data only.
when CRF=True, result is ~ 95% for F1,
but when I changed to CRF=False, the accuracy reduced to almost 0, wondering if I am doing something wrong:

Code:

def create_model(bert_config, is_training, input_ids, input_mask,
segment_ids, labels, num_labels, use_one_hot_embeddings,
dropout_rate=1.0, lstm_size=1, cell='lstm', num_layers=1):

  model = modeling.BertModel(
    config=bert_config,
    is_training=is_training,
    input_ids=input_ids,
    input_mask=input_mask,
    token_type_ids=segment_ids,
    use_one_hot_embeddings=use_one_hot_embeddings
)

embedding = model.get_sequence_output()
max_seq_length = embedding.shape[1].value

used = tf.sign(tf.abs(input_ids))
lengths = tf.reduce_sum(used, reduction_indices=1) 

blstm_crf = BLSTM_CRF(embedded_chars=embedding, hidden_unit=lstm_size, cell_type=cell, num_layers=num_layers, dropout_rate=dropout_rate, initializers=initializers, num_labels=num_labels,  seq_length=max_seq_length, labels=labels, lengths=lengths, is_training=is_training)

loss, logits, trans, pred_ids = blstm_crf.add_blstm_crf_layer(crf_only=False)

with tf.variable_scope("loss"):
    log_probs = tf.nn.log_softmax(logits, axis=-1)
    one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)
    per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
    loss = tf.reduce_sum(per_example_loss)
    probabilities = tf.nn.softmax(logits, axis=-1)
    predict = tf.argmax(probabilities,axis=-1)
    return (loss, per_example_loss, logits,predict)

Runing issue

Hi,

I tried to run ur code but it gave me this error

Can u help plz

thank u

run error

When running the code, I came across an error as follow:
Python version 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34) [GCC 7.3.0] PyTorch version 1.0.0 Current dir: /home/lxy/github/NER_BERT_CRF Cuda is available ? True Device: cuda:0 ***** Running training ***** Num examples = 14987 Batch size = 32 Num steps = 7025 100%|█████████████████████████████████| 213450/213450 [00:03<00:00, 59652.65B/s] *** Use only BertForTokenClassification *** 100%|█████████████████████████| 404400730/404400730 [05:49<00:00, 1157152.00B/s] Traceback (most recent call last): File "NER_BERT_CRF.py", line 484, in <module> for step, batch in enumerate(train_dataloader): File "/opt/anaconda/3-py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 637, in __next__ return self._process_next_batch(batch) File "/opt/anaconda/3-py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 658, in _process_next_batch raise batch.exc_type(batch.exc_msg) KeyError: 'Traceback (most recent call last):\n File "/opt/anaconda/3-py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop\n samples = collate_fn([dataset[i] for i in batch_indices])\n File "/opt/anaconda/3-py36/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in <listcomp>\n samples = collate_fn([dataset[i] for i in batch_indices])\n File "NER_BERT_CRF.py", line 304, in __getitem__\n feat = example2feature(self.examples[idx], self.tokenizer, self.label_map, max_seq_length)\n File "NER_BERT_CRF.py", line 248, in example2feature\n label_ids = [label_map[\'CLS\']]\nKeyError: \'CLS\'\n'

I could not figure out the error. What is wrong?

Is there a publication?

Hello,

Do you have a paper on this work? or planning to publish one?