something678/todkat

Transformer encoder-decoder for emotion detection in dialogues

TodKat

Transformer encoder-decoder for emotion detection in dialogues

Quick Start:

---

To run the model on test sets of four datasets,

1. Download the pre-trained models save.zip , unzip the zip file to ./TodKat/save
2. Locate to the ./TodKat/src and execute the following command in terminal: python TodKat_dd.py for DailyDialogue. python TodKat_emory.py for EmoryNLP. python TodKat_iemocap.py for IEMOCAP. python TodKat_meld.py for MELD.

Bug fix: the sklearn was used erroneously, causing the unusual high macro-F1s of MELD and EMORY. A hot fix is provided. Below lists the updated performance:

MELD		EMORY
avg-macro-F1	micro-F1	avg-macro-F1	micro-F1
0.6547	0.6724	0.3869	0.4238

To validate, simply download the saved model and run the code.

Here comes the detailed instructions on how to perform training, testing and prediction.

Required Packages:

---

torch==1.7.0 transformers==4.0.0 csv typing numpy tqdm pandas ipython ftfy==5.1

Run on GPU:

---

Model runs on GPU by default with cuda:0. Please change device if going to train on CPU. Saved models are trained with cuda:0, change the configurations in the saved model if you want to run them on another device.

Training:

---

Download the save.zip and put the 'topic-language-model-<dd|emory|iemocap|meld>' folder in the '../datasets/' directory. Uncomment TodKat_dd line 613 - line 705 to train model on DailyDialog. Uncomment TodKat_emory line 613 - line 705 to train model on EmoryNLP. Uncomment TodKat_iemocap line 614 - line 708 to train model on IEMOCAP. Uncomment TodKat_meld line 614 - line 708 to train model on MELD.

Evaluation:

---

Comment TodKat_dd line 613 - line 705 and uncomment line 707 - 738 on DailyDialog. Comment TodKat_emory line 613 - line 705 and uncomment line 707 - 744 on EmoryNLP. Comment TodKat_iemocap line 614 - line 708 and uncomment line 710 - 741 on IEMOCAP. Comment TodKat_meld line 614 - line 708 and uncomment line 710 - 742 on MELD.

Prediction:

---

Comment TodKat_dd line 613 - line 705 and uncomment line 707 - 742 on DailyDialog. Comment TodKat_emory line 613 - line 705 and uncomment line 707 - 748 on EmoryNLP. Comment TodKat_iemocap line 614 - line 708 and uncomment line 710 - 745 on IEMOCAP. Comment TodKat_meld line 614 - line 708 and uncomment line 710 - 746 on MELD.

todkat's Issues

wrong results on emorynlp

Thanks for your nice work for erc, in the file src/DialogEvaluator_emory.py line 145, macro f1 only contains 6 labels, without the neutral

        macro_f1 = metrics.f1_score(
            pred_list,
            grd_list,
            labels=[1, 2, 3, 4, 5, 6],
            average='weighted')
        logging.info("Weighted F1-macro with neutral: {:.4f} ({}/{})\n".format(macro_f1, correct, total))

labels here should be [0, 1, 2, 3, 4, 5, 6]

I wonder if there's any code about the topic representation learning. It would be a great help if you can realease this part of code.😊

Codes for commonsense selection and integration

Hello, authors, thank you for your impressive work ! However, the corresponding codes for commonsense selection and integration seem to be missing, could you please update them?
Thank you very much !

RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

  File "TodKat_meld.py", line 697, in <module>
    modelDlg.fit(
  File "TodKat_meld.py", line 428, in fit
    tfboyed_features = model(features)
  File "/home/phd-fan.weiquan2/anaconda3/envs/torch17py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/phd-fan.weiquan2/anaconda3/envs/torch17py38/lib/python3.8/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/home/phd-fan.weiquan2/anaconda3/envs/torch17py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/phd-fan.weiquan2/works/MMMTD_Benchmark/TodKat/src/transformerunit.py", line 41, in forward
    uttrs_tr = self.transformerlayer(uttrs)
  File "/home/phd-fan.weiquan2/anaconda3/envs/torch17py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/phd-fan.weiquan2/anaconda3/envs/torch17py38/lib/python3.8/site-packages/torch/nn/modules/transformer.py", line 293, in forward
    src2 = self.self_attn(src, src, src, attn_mask=src_mask,
  File "/home/phd-fan.weiquan2/anaconda3/envs/torch17py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/phd-fan.weiquan2/anaconda3/envs/torch17py38/lib/python3.8/site-packages/torch/nn/modules/activation.py", line 978, in forward
    return F.multi_head_attention_forward(
  File "/home/phd-fan.weiquan2/anaconda3/envs/torch17py38/lib/python3.8/site-packages/torch/nn/functional.py", line 4146, in multi_head_attention_forward
    q, k, v = linear(query, in_proj_weight, in_proj_bias).chunk(3, dim=-1)
  File "/home/phd-fan.weiquan2/anaconda3/envs/torch17py38/lib/python3.8/site-packages/torch/nn/functional.py", line 1692, in linear
    output = input.matmul(weight.t())
RuntimeError: Tensor for 'out' is on CPU, Tensor for argument #1 'self' is on CPU, but expected them to be on GPU (while checking arguments for addmm)

I wonder why this quenstion occurs, thank you.

Can't access"https://huggingface.co/api/save/topic-language-model-emory"

while runningpython TodKat_emory.py, error turns out :
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/save/topic-language-model-emory (Request ID: bQk4-y5w0U6_fydjXfwr9)

model path error

I'm trying to run the training code. So I uncommented the lines and ran the code.
I got the following error:

After checking the cause, model_path is set as follows.
tokenizer_name = '../datasets/topic-language-model-iemocap'

Is this the intended code? There is no folder on github for the model. Isn't RoBERTa's pretraeind path roberta-{size}?

thank you.

Error with meld dataset

Hello,
I was wondering if something changed for the meld dataset? See this 404 error I got while trying to run the model:

404 Client Error: Not Found for url: https://huggingface.co/datasets/topic-language-model-meld/resolve/main/config.json

OSError: Can't load config for '../datasets/topic-language-model-meld'. Make sure that:

• '../datasets/topic-language-model-meld' is a correct model identifier listed on 'https://huggingface.co/models'

• or '../datasets/topic-language-model-meld' is the correct path to a directory containing a config.json file

Any information would be very appreciated,
Thank you :)

Taylor

Wrong Weighted-Avg F1 in All Experiments

Hi, many thanks for your great work on ToDKat. It helps us a lot.
However, I found that you made mistakes on all weighted F1 calculations in all experiments via metrics.f1_score, e.g.

TodKat/src/DialogEvaluator_meld.py

Line 145 in abf6a13

macro_f1 = metrics.f1_score(

In scikit doc, f1_score takes y_true first, then y_pred. And the weights is based on y_true. In your exps, you put pred_list first...
I downloaded your pretrained weights, and if this bug is fixed, the weighted F1 drops from 68.23 to 61.28.

Logs for reference:
Before fixing the bug:
2021-12-16 11:46:30 - Accuracy: 0.6475 (1690/2610)
2021-12-16 11:46:30 - Weighted F1-macro with neutral: 0.6823 (1690/2610)
2021-12-16 11:46:30 - F1-micro with neutral: 0.6475 (1690/2610)

After fixing the bug:
2021-12-16 11:48:32 - Accuracy: 0.6475 (1690/2610)
2021-12-16 11:48:32 - Weighted F1-macro with neutral: 0.6128 (1690/2610)
2021-12-16 11:48:32 - F1-micro with neutral: 0.6475 (1690/2610)

Where is the topic vector based on VAE

I can not find the topic vector code based on VAE. Would someone find it?

Where is VAE code???

Cuda Issue while training.

Followed the steps as mentioned in the ReadMe. Still facing error while training the IEMOCAP model.
Some CPU/GPU issue I suppose. Would be great if it could be resolved.
Attaching the pic of the error.

Knowledge Graph

How to struct Knowledge Graph? I want to know how this part is implemented

There is only "self.transformerlayer = TransformerEncoderLayer" in "class TransformerUnit", but no "TransformerDecoderLayer".

Hi there, I found TransformerDecoderLayer is not used in the code repo, and I wonder why this is different from the Transformer Encoder-Decoder Classifier in the paper. Thanks!!!

who could tell me where is tramsfomers file？

TodKat/src/roberta_with_finetune.py

Line 4 in 1f71149

from transformers import RobertaModel, RobertaTokenizer

Where is the point network？

Hello, authors,

First of all, Thank you for your interesting work!

I note you have mentioned in your paper that you utilize the point network to select the knowledge coming from the pre-trained model. I need this part of the function, however, I can't find this corresponding code in your repository.

Best!