sdadas/polish-roberta

RoBERTa models for Polish

polish-roberta's Issues

Training time

How long did it take to train the base and large models?

Problem with padding

I would like to use RobertaForSequenceClassification(huggingface) with your model, but I have a problem with padding. For BERT, I would use tokenizer.encode_plus() to deal with special tokens and padding. Here we have SentencePieceBPETokenizer, which doesn't have such method.
I tried to use tokenizer.enable_padding(max_length = 64) and after encoding I get smth like [0, 50, 15,17, 20,11,2, 0, ... , 0]. Though tokenizer.token_to_id('<pad>') = 1.
Also, when creating model object, i see that padding_idx=1.

RobertaForSequenceClassification( 
(roberta): RobertaModel(
(embeddings): RobertaEmbeddings(
(word_embeddings): Embedding(50001, 768, padding_idx=1)
(position_embeddings): Embedding(514, 768, padding_idx=1)
(token_type_embeddings): Embedding(1, 768)
(LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(dropout): Dropout(p=0.1, inplace=False))
...

I don't know if there is a problem with it.
According to the results, the model works fine and I got pretty good results.

p.s. I appriciate your work

Transformers models

There is a problem with the base model. In vocab.json an entry with id 50000 is missing.

In the large model a vocab size is 50009, but in vocab.json is no entry with id 50008 - however it does not cause errors.

File missing in data

I tried to replicate your experiments. I downloaded the data as per instructions, using download_data.py. However, when trying to run run_tasks.py, an error occurs: "No such file or directory: 'data_processed/KLEJ/NKJP-NER-bin/input0/dict.txt'".

Thanks for your help!

Publication with details of training?

Do you plan to publish the details of the training process? The results are excellent and it would be very beneficial for the research community to know the details of training.

Distilroberta Huggingface usage for embeddings

Cześć :). I'd like to use this model https://huggingface.co/sdadas/st-polish-paraphrase-from-distilroberta for embeddings, ideally using huggingface automatic api generation feature :) I'm getting "string indices must be integers" error from the api, which I assume means that it's not configured and it always uses the sentence similarity api. I'd be very happy to implement handler for this and contribute it to this repo - but I'd need some guidance.

missing 1 required positional argument: 'tokenizer'

I get the following error when running tokenizer = PreTrainedTokenizerFast(tokenizer_file="roberta_large_transformers/tokenizer.json"):

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() missing 1 required positional argument: 'tokenizer

Do you have any idea to solve it? By the way, I prefer to load the tokenizer by something like:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("roberta_large_transformers")

I tried such code but I failed to load that and getting the error:

  File "/home/yongjiang.jy/wangxy/flair/flair/embeddings.py", line 2927, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(model, **kwargs)
  File "/home/yongjiang.jy/anaconda2/envs/parser/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 217, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/yongjiang.jy/anaconda2/envs/parser/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1140, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home/yongjiang.jy/anaconda2/envs/parser/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 1287, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/yongjiang.jy/anaconda2/envs/parser/lib/python3.6/site-packages/transformers/tokenization_roberta.py", line 162, in __init__
    **kwargs,
  File "/home/yongjiang.jy/anaconda2/envs/parser/lib/python3.6/site-packages/transformers/tokenization_gpt2.py", line 157, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType

It seems that the vocab.json is missing, do you have any idea about this?

Thank you!

Problem with running example from README.md

After downloading data using
python download_data.py
command and downloading and uzipping model to roberta_base_fairseq directory I tried to run
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks KLEJ-NKJP --fp16 True --max-sentences 8 --update-freq 2
command, but i got the following error:

File "/home/torales/.local/lib/python3.7/site-packages/fairseq/checkpoint_utils.py", line 189, in load_model_ensemble_and_task raise IOError("Model file not found: {}".format(filename)) OSError: Model file not found: checkpoints/roberta_base_fairseq/KLEJ/NKJP-NER/checkpoint_best.pt

Any ideas how can I fix it?