Giter Site home page Giter Site logo

xlm-t's Introduction

This is the XLM-T repository, which includes data, code and pre-trained multilingual language models for Twitter.

XLM-T - A Multilingual Language Model Toolkit for Twitter

As explained in the reference paper, we make start from XLM-Roberta base and continue pre-training on a large corpus of Twitter in multiple languages. This masked language model, which we named twitter-xlm-roberta-base in the 🤗Huggingface hub, can be downloaded from here.

Note: This Twitter-specific pretrained LM was pretrained following a similar strategy to its English-only counterpart, which was introduced as part of the TweetEval framework, and available here. All models, including this multilingual one, have been integrated into the TweetNLP framework.

We also provide task-specific models based on the Adapter technique, fine-tuned for cross-lingual sentiment analysis (See #3):

1 - Models

We release the following multilingual language models in Hugging Face:

  • XLM-T Language Model: XLM-R language model trained on 200M tweets for 30+ languages.

  • XLM-T Sentiment: The same XLM model above fine-tuned on our UMSAB multilingual sentiment analysis dataset (see #3).

2 - Code

We include code with various functionalities to complement this release. We provide examples for, among others, feature extraction and adapter-based inference with language models in this notebook. Also with examples for training and evaluating language models on multiple tweet classification tasks, compatible with UMSAB (see #3) and TweetEval datasets.

Perform inference with Huggingface's pipelines

Using Huggingface's pipelines, obtaining predictions is as easy as:

from transformers import pipeline
model_path = "cardiffnlp/twitter-xlm-roberta-base-sentiment"
sentiment_task = pipeline("sentiment-analysis", model=model_path, tokenizer=model_path)
sentiment_task("Huggingface es lo mejor! Awesome library 🤗😎")
[{'label': 'Positive', 'score': 0.9343640804290771}]

Fine-tune xlm-t with adapters

You can fine-tune an adapter built on top of your language model of choice by running the src/adapter_finetuning.py script, for example:

python3 src/adapter_finetuning.py --language spanish --model cardfiffnlp/twitter-xlm-roberta-base --seed 1 --lr 0.0001 --max_epochs 20

Notebooks

For quick prototyping, you can direclty use the Colab notebooks we provide below:

Notebook Description Colab Link
01: Playgroud examples Minimal start examples Open In Colab
02: Extract embeddings Extract embeddings from tweets Open In Colab
03: Sentiment prediction Predict sentiment Open In Colab
04: Fine-tuning Fine-tune a model on custom data Open In Colab

3 - UMSAB, the Unified Multilingual Sentiment Analysis Benchmark

As part of our framework, we also release a unified benchmark for cross-lingual sentiment analysis for eight different languages. All datasets are framed as tweet classification with three labels (positive, negative and neutral). The languages included in the benchmark, as well as the datasets they are based on, are: Arabic (SemEval-2017, Rosenthal et al. 2017), English (SemEval-17, Rosenthal et al. 2017), French (Deft-2017, Benamara et al. 2017), German (SB-10K, Cieliebak et al. 2017), Hindi (SAIL 2015, Patra et al. 2015), Italian (Sentipolc-2016, Barbieri et al. 2016), Portuguese (SentiBR, Brum and Nunes, 2017) and Spanish (InterTASS 2017, Díaz Galiano et al. 2018). The format for each dataset follows that of TweetEval with one line per tweet and label per line.

UMSAB Results / Leaderboard

The following results (Macro F1 reported) correspond to XLM-R (Conneau et al. 2020) and XLM-Tw, the same model retrained on Twitter as explained in the reference paper. The two settings are monolingual (trained and tested in the same language) and multilingual (considering all languages for training). Check the reference paper for more details on the setting and the metrics.

FT Mono XLM-R Mono XLM-Tw Mono XLM-R Multi XLM-Tw Multi
Arabic 46.0 63.6 67.7 64.3 66.9
English 50.9 68.2 66.9 68.5 70.6
French 54.8 72.0 68.2 70.5 71.2
German 59.6 73.6 76.1 72.8 77.3
Hindi 37.1 36.6 40.3 53.4 56.4
Italian 54.7 71.5 70.9 68.6 69.1
Portuguese 55.1 67.1 76.0 69.8 75.4
Spanish 50.1 65.9 68.5 66.0 67.9
All lang. 51.0 64.8 66.8 66.8 69.4

If you would like to have your results added to the leaderboard you can either submit a pull request or send an email to any of the paper authors with results and the predictions of your model. Please also submit a reference to a paper describing your approach.

Evaluating your system

For evaluating your system according to Macro-F1, you simply need an individual prediction file for each of the languages. The format of the predictions file should be the same as the output examples in the predictions folder (one output label per line as per the original test file) and the files should be named language.txt (e.g. arabic.txt or all.txt if evaluating all languages at once). The predictions included as an example in this repo correspond to xlm-t trained and evaluated on all languages (All lang.).

Example usage

python src/evaluation_script.py

The script takes as input a set of test labels and the predictions from the "predictions" folder by default, but you can set this to suit your needs as optional arguments.

Optional arguments

Three optional arguments can be modified:

--gold_path: Path to gold datasets. Default: ./data/sentiment

--predictions_path: Path to predictions directory. Default: ./predictions/sentiment

--language: Language to evaluate (arabic, english ... or all). Default: all

Evaluation script sample usage from the terminal with parameters:

python src/evaluation_script.py --gold_path ./data/sentiment --predictions_path ./predictions/sentiment --language arabic

(this script would output the results for the Arabic dataset only)

Reference paper

If you use this repository in your research, please use the following bib entry to cite the reference paper.

@InProceedings{barbieri-espinosaanke-camachocollados:2022:LREC,
  author    = {Barbieri, Francesco  and  Espinosa Anke, Luis  and  Camacho-Collados, Jose},
  title     = {XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond},
  booktitle      = {Proceedings of the Language Resources and Evaluation Conference},
  month          = {June},
  year           = {2022},
  address        = {Marseille, France},
  publisher      = {European Language Resources Association},
  pages     = {258--266},
  abstract  = {Language models are ubiquitous in current NLP, and their multilingual capacity has recently attracted considerable attention. However, current analyses have almost exclusively focused on (multilingual variants of) standard benchmarks, and have relied on clean pre-training and task-specific corpora as multilingual signals. In this paper, we introduce XLM-T, a model to train and evaluate multilingual language models in Twitter. In this paper we provide: (1) a new strong multilingual baseline consisting of an XLM-R (Conneau et al. 2020) model pre-trained on millions of tweets in over thirty languages, alongside starter code to subsequently fine-tune on a target task; and (2) a set of unified sentiment analysis Twitter datasets in eight different languages and a XLM-T model trained on this dataset.},
  url       = {https://aclanthology.org/2022.lrec-1.27}
}

If using UMSAB, please also cite their corresponding datasets.

License

This repository is released open-source but but restrictions may apply to individual datasets (which are derived from existing data) or Twitter (main data source). We refer users to the original licenses accompanying each dataset and Twitter regulations.

xlm-t's People

Contributors

luisespinosaanke avatar pedrada88 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

xlm-t's Issues

Distilled or smaller pretrained models?

Hi

xlm-t works really well on my dataset however to make it effective performance-wise I was wondering if you guys are planning to release smaller pretrained or distilled models?

Best
Anup

Better preprocessing

Hello,

I was wondering if the preprocess function could be enhanced as right now, it strips punctuations before and after usernames/URLs. Or was it done on purpose? I couldn't find a justification of this in your paper.

Right now, the preprocess function below would convert:

I love you @louisia!!!!

to

I love you @user

# Preprocess text (username and link placeholders)
def preprocess(text):
    new_text = []
    for t in text.split(" "):
        t = '@user' if t.startswith('@') and len(t) > 1 else t
        t = 'http' if t.startswith('http') else t
        new_text.append(t)
    return " ".join(new_text)

It seems to me that punctuations could help the model predict the sentiment of a tweet a little better if it was available to it. Another example: some users on twitter, start their tweets with a dot like this:

.@rudy is really bad. What a shame.

They do that to avoid the reply system while still quoting a username. With the actual pre-processing function, "@rudy" doesn't get replaced because there is a dot right before the @.

Is there any particular reason why the preprocessing function was done this way or we could try to make it more flexible in our end by keeping the punctuations next to usernames or URLs?

Thank you!

Output labels

How could I get all three possible labels with probabilities in output?

Issues with finetuning colab notebook

The fine-tuning colab notebook doesn't fully work.

The first cell in the Fine-tuning section errors out. I'm guessing it's probably due to specific versions of packages being required (which are not enforced in the first cell).

Also, as a very minor additional thing, this link is broken: This notebook was modified from https://huggingface.co/transformers/custom_datasets.html.

ValueError: You have to specify either input_ids or inputs_embeds

ubuntu16.04
adapter-transformers==1.1.1
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:81:00.0 Off | N/A |
| 41% 26C P8 20W / 250W | 0MiB / 11019MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

when I run adapter_fintuning.py I got this error:

root@ubuntu:/home/project/xlm-t-main# python src/adapter_finetuning.py
Some weights of the model checkpoint at cardiffnlp/twitter-xlm-roberta-base were not used when initializing XLMRobertaModelWithHeads: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']

  • This IS expected if you are initializing XLMRobertaModelWithHeads from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
  • This IS NOT expected if you are initializing XLMRobertaModelWithHeads from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
    Some weights of XLMRobertaModelWithHeads were not initialized from the model checkpoint at cardiffnlp/twitter-xlm-roberta-base and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
    You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
    0%| | 0/200 [00:00<?, ?it/s]Traceback (most recent call last):
    File "src/adapter_finetuning.py", line 157, in
    trainer.train()
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/transformers/trainer.py", line 787, in train
    tr_loss += self.training_step(model, inputs)
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/transformers/trainer.py", line 1138, in training_step
    loss = self.compute_loss(model, inputs)
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/transformers/trainer.py", line 1162, in compute_loss
    outputs = model(**inputs)
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 805, in forward
    return_dict=return_dict,
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 685, in forward
    raise ValueError("You have to specify either input_ids or inputs_embeds")
    ValueError: You have to specify either input_ids or inputs_embeds
    0%| | 0/200 [00:00<?, ?it/s]
    (train_cpu) root@ubuntu:/home/project/xlm-t-main# python src/adapter_finetuning.py
    Some weights of the model checkpoint at cardiffnlp/twitter-xlm-roberta-base were not used when initializing XLMRobertaModelWithHeads: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']
  • This IS expected if you are initializing XLMRobertaModelWithHeads from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
  • This IS NOT expected if you are initializing XLMRobertaModelWithHeads from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
    Some weights of XLMRobertaModelWithHeads were not initialized from the model checkpoint at cardiffnlp/twitter-xlm-roberta-base and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
    You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
    0%| | 0/200 [00:00<?, ?it/s]Traceback (most recent call last):
    File "src/adapter_finetuning.py", line 157, in
    trainer.train()
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/transformers/trainer.py", line 787, in train
    tr_loss += self.training_step(model, inputs)
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/transformers/trainer.py", line 1138, in training_step
    loss = self.compute_loss(model, inputs)
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/transformers/trainer.py", line 1162, in compute_loss
    outputs = model(**inputs)
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 805, in forward
    return_dict=return_dict,
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
    File "/root/anaconda3/envs/train_cpu/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 685, in forward
    raise ValueError("You have to specify either input_ids or inputs_embeds")
    ValueError: You have to specify either input_ids or inputs_embeds
    0%| | 0/200 [00:00<?, ?it/s]

anybody can help?

Run on Colab CRASH

I follow the step to run the model using a dataset with 5k tweets. However, when doing the 'output = model(**encoded_input)' the colab pro+ always crashes. Is there any solution to this issue? Thanks a lot!!!!

Use models with zero-shot classification

Hello,

First, thanks for these great models! I was wondering if I could use these models for zero-shots classification, especially for emotion detection (Ekman). While doing so, I encountered this error regarding the config of the models and the missing entailment parameter which seems mandatory for zero-shot classification:

Failed to determine 'entailment' label id from the label2id mapping in the model config. Setting to -1. Define a descriptive label2id mapping in the model config to ensure correct outputs.

Basic code to reproduce:

model_path = "cardiffnlp/twitter-xlm-roberta-base"
zero_shot = pipeline("zero-shot-classification", model=model_path, tokenizer=model_path,)
zero_shot("You are an awful man,", candidate_labels=["anger", "disgust", "fear", "joy", "sadness", "surprise"], multi_label=True,)

It seems to lead indeed to weird outputs compared to the XLM-Roberta XNLI model that can be tested here: https://huggingface.co/zero-shot/.

Is there something missing in head_config.json?

Thanks!

AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'

Hi
Im using transformers version '4.26.1' on databricks. The code below return the following error

Code:

model_path = "cardiffnlp/twitter-xlm-roberta-base-sentiment"
sentiment_task = pipeline("sentiment-analysis", model=model_path, tokenizer=model_path)

Error:

AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<command-798773577171854> in <module>
     15 
     16 model_path = "cardiffnlp/twitter-xlm-roberta-base-sentiment"
---> 17 sentiment_task = pipeline("sentiment-analysis", model=model_path, tokenizer=model_path)
     18 
     19 df_group_notna['sentiment'] = df_group_notna['Feedback'].apply(lambda x: sentiment_task(x)[0]['label'])

/databricks/python/lib/python3.7/site-packages/transformers/pipelines/__init__.py in pipeline(task, model, config, tokenizer, feature_extractor, framework, revision, use_fast, use_auth_token, device, device_map, torch_dtype, trust_remote_code, model_kwargs, pipeline_class, **kwargs)
    827 
    828             tokenizer = AutoTokenizer.from_pretrained(
--> 829                 tokenizer_identifier, use_fast=use_fast, _from_pipeline=task, **hub_kwargs, **tokenizer_kwargs
    830             )
    831 
/databricks/python/lib/python3.7/site-packages/transformers/models/auto/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    674             tokenizer_class_py, tokenizer_class_fast = TOKENIZER_MAPPING[type(config)]
    675             if tokenizer_class_fast and (use_fast or tokenizer_class_py is None):
--> 676                 return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    677             else:
    678                 if tokenizer_class_py is not None:

/databricks/python/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1811             local_files_only=local_files_only,
   1812             _commit_hash=commit_hash,
-> 1813             **kwargs,
   1814         )
   1815 

/databricks/python/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in _from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, use_auth_token, cache_dir, local_files_only, _commit_hash, *init_inputs, **kwargs)
   1957         # Instantiate tokenizer.
   1958         try:
-> 1959             tokenizer = cls(*init_inputs, **init_kwargs)
   1960         except OSError:
   1961             raise OSError(

/databricks/python/lib/python3.7/site-packages/transformers/models/xlm_roberta/tokenization_xlm_roberta_fast.py in __init__(self, vocab_file, tokenizer_file, bos_token, eos_token, sep_token, cls_token, unk_token, pad_token, mask_token, **kwargs)
    163             pad_token=pad_token,
    164             mask_token=mask_token,
--> 165             **kwargs,
    166         )
    167 

/databricks/python/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py in __init__(self, *args, **kwargs)
    112         elif slow_tokenizer is not None:
    113             # We need to convert a slow tokenizer to build the backend
--> 114             fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
    115         elif self.slow_tokenizer_class is not None:
    116             # We need to create and convert a slow tokenizer to build the backend

/databricks/python/lib/python3.7/site-packages/transformers/convert_slow_tokenizer.py in convert_slow_tokenizer(transformer_tokenizer)
   1160     converter_class = SLOW_TO_FAST_CONVERTERS[tokenizer_class_name]
   1161 
-> 1162     return converter_class(transformer_tokenizer).converted()

/databricks/python/lib/python3.7/site-packages/transformers/convert_slow_tokenizer.py in __init__(self, *args)
    436         super().__init__(*args)
    437 
--> 438         from .utils import sentencepiece_model_pb2 as model_pb2
    439 
    440         m = model_pb2.ModelProto()

/databricks/python/lib/python3.7/site-packages/transformers/utils/sentencepiece_model_pb2.py in <module>
     32     syntax="proto2",
     33     serialized_options=b"H\003",
---> 34     create_key=_descriptor._internal_create_key,
     35     serialized_pb=(
     36         b'\n\x19sentencepiece_model.proto\x12\rsentencepiece"\xa1\n\n\x0bTrainerSpec\x12\r\n\x05input\x18\x01'

AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.