Loss calculation error

[huggingface]

Loss calculation error

[thomwolf]

Hi Jian, can you give me a small (self-contained) example showing how to get this error?

[jwang-lp]

Hi Thomas! I modified the code in your README.md for an example:

from pytorch_pretrained_bert.modeling import BertForMaskedLM, BertConfig
from pytorch_pretrained_bert import BertTokenizer
import torch

model = BertForMaskedLM.from_pretrained('bert-base-uncased')

# Tokenized input
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = "Who was Jim Henson ? Jim Henson was a puppeteer"
tokenized_text = tokenizer.tokenize(text)

# Mask a token that we will try to predict back with `BertForMaskedLM`
masked_index = 6
tokenized_text[masked_index] = '[MASK]'

# Convert token to vocabulary indices
indexed_truths = tokenizer.convert_tokens_to_ids(tokenized_text)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
indexed_truths_tensor = torch.tensor([indexed_truths])

# Evaluate loss
model.eval()
masked_lm_logits_scores = model(tokens_tensor, masked_lm_labels=indexed_truths_tensor)
print(masked_lm_logits_scores)

[thomwolf]

Thank you, you are right, I fixed that on master. It will be in the next release.