hltchkust/paml

Personalizing Dialogue Agents via Meta-Learning

PAML Personalizing Dialogue Agents via Meta-Learning

This is the PyTorch implementation of the paper: Personalizing Dialogue Agents via Meta-Learning. Zhaojiang Lin*, Andrea Madotto*, Chien-Sheng Wu, Pascale Fung ACL 2019 [PDF]

Zhaojiang Lin and Andrea Madotto contributed equally to this work.

This code has been written using PyTorch >= 0.4.1. If you use any source codes or datasets included in this toolkit in your work, please cite the following paper. The bibtex is listed below:

@article{lin2019personalizing,
  title={Personalizing Dialogue Agents via Meta-Learning},
  author={Lin, Zhaojiang and Madotto, Andrea and Wu, Chien-Sheng and Fung, Pascale},
  journal={arXiv preprint arXiv:1905.10033},
  year={2019}
}

Abstract

Existing personalized dialogue models use human designed persona descriptions to improve dialogue consistency. Collecting such descriptions from existing dialogues is expensive and requires hand-crafted feature designs. In this paper, we propose to extend Model-Agnostic Meta-Learning (MAML) to personalized dialogue learning without using any persona descriptions. Our model learns to quickly adapt to new personas by leveraging only a few dialogue samples collected from the same user, which is fundamentally different from conditioning the response on the persona descriptions. Empirical results on Persona-chat dataset indicate that our solution outperforms non-meta-learning baselines using automatic evaluation metrics, and in terms of human-evaluated fluency and consistency.

Persona-agnostic meta-learning

The difference between finetuning from a) joint training on all personas and b) meta-learning persona. The solid line represents the optimization path of the initial parameters and dashed line the fine-tuning path. Meta-learned initial parameters can faster adapt to a new persona.

Consistency improvement

Iteration of finetuning versus consistency. Consistency of PAML grows linearly with respect to the iteration.

K-shot (10 iteration) results for different settings. Consistency of PAML grows linearly with respect to finetune dialogue number.

Dependency

Check the packages needed or simply run the command

❱❱❱ pip install -r requirements.txt

Pre-trained glove embedding: glove.6B.300d.txt inside folder /vectors/.

Trained NLI model discribed in paper 3.1: pytorch_model.bin inside folder /data/nli_model/.

Experiment

Training

Training PAML

❱❱❱ python MAML.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --fix_dialnum_train --pointer_gen --save_path save/paml/

Training baseline without persona input

❱❱❱ python main.py --cuda --model trs --pretrain_emb --weight_sharing --label_smoothing --noam --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/no_persona/ 

Training baseline with persona input

❱❱❱ python main.py --cuda --model trs --pretrain_emb --weight_sharing --label_smoothing --noam --emb_dim 300 --hidden_dim 300 --pointer_gen --persona --save_path save/persona/

After training, take the model with lowest PPL in validation set to finetune and test (replace ${model}).

Automatic metric

Finetune and Testing PAML

❱❱❱ python main_fine_tune.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/paml/${model} --save_path_dataset save/paml/ --test

Finetune and Testing baseline without persona input

❱❱❱ python main_fine_tune.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/no_persona/${model} --save_path_dataset save/no_persona/ --test

Finetune and Testing baseline with persona input

❱❱❱ python main_fine_tune.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --pointer_gen --persona --save_path save/persona/${model} --save_path_dataset save/persona/ --test

Generation samples

To check generation of PAML:

❱❱❱ python generate_samples.py --cuda --model trs --batch_size 1 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/paml/${model} --save_path_dataset save/paml/ --test

To skip training and finetune, check generation here: paml_generation

paml's Issues

function Lang lost

vocab = Lang(),fucntion Lang () is lost

Need help on reproducing the results.

• I followed the instructions in README to build up the environments and successfully got the codes run.
• Using the script python MAML.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --fix_dialnum_train --pointer_gen --save_path save/paml/, I trained the model for several hours and it stopped automatically. And I got a good checkpoint save/paml/model_1_50.1856_0.0000_0.0000_0.0000_1.1000.
• And I run python main_fine_tune.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/paml/model_1_50.1856_0.0000_0.0000_0.0000_1.1000 --save_path_dataset save/paml/ --test to have a test. Unexpectedly, I get the following outputs:

['hi there how are you doing ?',
 'i am doing good . just enjoying eating my muffin , you ?',
 'i am well thank you just waiting for my kiddos to get home from school . '
 'than dinner',
 'i wish i had kids . just me and my dog , pedro , here',
 'give it time dear . they are a lot of work and well worth the wait',
 'i work with a lot of kids in the healthcare industry',
 'that is a noble profession how did you get into that',
 'i originally wanted to have a career in film as i love it',
 'what exactly do you want to do in that field ?',
 'produce . i decided to go down the healthcare route because its hard to get '
 'into',
 'i can see that do you have a starting place ?',
 'i am not sure yet . so , what do you like ?',
 'reading is a passion , as well as cooking for my family . pretty simple here',
 'sounds like a nice life you got',
 'i think it is . everyday life with little drama from outsiders just how i '
 'like it']
Beam: i am a i am i am as as as as as as as ?
Ref:well , i am happy for you !
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi there how are you doing ?',
 'i am doing good . just enjoying eating my muffin , you ?',
 'i am well thank you just waiting for my kiddos to get home from school . '
 'than dinner',
 'i wish i had kids . just me and my dog , pedro , here',
 'give it time dear . they are a lot of work and well worth the wait',
 'i work with a lot of kids in the healthcare industry',
 'that is a noble profession how did you get into that',
 'i originally wanted to have a career in film as i love it',
 'what exactly do you want to do in that field ?',
 'produce . i decided to go down the healthcare route because its hard to get '
 'into',
 'i can see that do you have a starting place ?',
 'i am not sure yet . so , what do you like ?',
 'reading is a passion , as well as cooking for my family . pretty simple here']
Beam: i am as as as as as as as as as as as as as as as
Ref:sounds like a nice life you got
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi there how are you doing ?',
 'i am doing good . just enjoying eating my muffin , you ?',
 'i am well thank you just waiting for my kiddos to get home from school . '
 'than dinner',
 'i wish i had kids . just me and my dog , pedro , here',
 'give it time dear . they are a lot of work and well worth the wait',
 'i work with a lot of kids in the healthcare industry',
 'that is a noble profession how did you get into that',
 'i originally wanted to have a career in film as i love it',
 'what exactly do you want to do in that field ?',
 'produce . i decided to go down the healthcare route because its hard to get '
 'into',
 'i can see that do you have a starting place ?']
Beam: i am ? i am i am i am i am as as as as
Ref:i am not sure yet . so , what do you like ?
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi there how are you doing ?',
 'i am doing good . just enjoying eating my muffin , you ?',
 'i am well thank you just waiting for my kiddos to get home from school . '
 'than dinner',
 'i wish i had kids . just me and my dog , pedro , here',
 'give it time dear . they are a lot of work and well worth the wait',
 'i work with a lot of kids in the healthcare industry',
 'that is a noble profession how did you get into that',
 'i originally wanted to have a career in film as i love it',
 'what exactly do you want to do in that field ?']
Beam: i am i am as as as as as as as as as as as as as
Ref:produce . i decided to go down the healthcare route because its hard to get into
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi there how are you doing ?',
 'i am doing good . just enjoying eating my muffin , you ?',
 'i am well thank you just waiting for my kiddos to get home from school . '
 'than dinner',
 'i wish i had kids . just me and my dog , pedro , here',
 'give it time dear . they are a lot of work and well worth the wait',
 'i work with a lot of kids in the healthcare industry',
 'that is a noble profession how did you get into that']
Beam: i am ? i am i am i am i am i am i am series
Ref:i originally wanted to have a career in film as i love it
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi there how are you doing ?',
 'i am doing good . just enjoying eating my muffin , you ?',
 'i am well thank you just waiting for my kiddos to get home from school . '
 'than dinner',
 'i wish i had kids . just me and my dog , pedro , here',
 'give it time dear . they are a lot of work and well worth the wait']
Beam: i am ? i am a i am i am i am i am i am acting
Ref:i work with a lot of kids in the healthcare industry
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi there how are you doing ?',
 'i am doing good . just enjoying eating my muffin , you ?',
 'i am well thank you just waiting for my kiddos to get home from school . '
 'than dinner']
Beam: i am ? i am i am i am i am as as as as
Ref:i wish i had kids . just me and my dog , pedro , here
----------------------------------------------------------------------
----------------------------------------------------------------------
['hi there how are you doing ?']
Beam: i am pass especially a i am especially a i am do half the
Ref:i am doing good . just enjoying eating my muffin , you ?
----------------------------------------------------------------------
----------------------------------------------------------------------
EVAL	Loss	Peplexity	Entl_b	Bleu_b
test	6.6667	785.7816	0.00	0.00

After several fine-tuning steps (100 iterations?), it stops and it does not seem to get better:

----------------------------------------------------------------------
{'Bleu_b': array([0.        , 0.00462254, 0.        , 0.00353671, 0.0037849 ,
       0.00433299]), 'Entl_b': array([-0.10235387, -0.02899247, -0.00670951,  0.00621429,  0.01460281,
        0.01853968]), 'PPL': array([6289.55504171, 1681.21756928, 1077.9117038 ,  916.79228123,
        836.58960702,  767.79717897]), 'LOSS': array([8.62963292, 7.31024756, 6.876336  , 6.71580096, 6.62371861,
       6.53619826])}
----------------------------------------------------------------------

• I haven't gone deeper for the codes. But I'd like to build up your dialog system quickly and do some brainstorms after observation. Could you please help me find out the problems in my reproduction?
• And will you share your checkpoints to reproduce the results. It will be great!

Meta-Evaluation

Hello
Thanks for great work !

In MAML.py line 172 the Meta-Evaluation starts

My question is that would this part have any impact on the performance of the model.

I think the answer is YES, because in line 190 the function do_learning_fix_step calls model.train_one_batch in line 81 which computes grad of loss and update parameters via optimizer.step() here

However, the grad update and back propagation procedures had already been taken
(i) for train batch at line 150 and (ii) for valid batch at line 170 and hence I believe that there should be no extra grad update and weight update

Therefore I think the answer to my question should be NO, implying that there should be no more grad update during Meta-Evaluation. If your intention for Meta-Evaluation was to print eval loss and ppl, and also save model, then i think line 190 is inappropriate and needs to be fixed such as giving a train=False flag for train_one_batch function at least during Meta-Evaluation

Thanks in advance

Applying F.log_softmax on Generator otuput

Hi

You have applied F.log_sofmax on the output of projection layer in [line 232] (

PAML/model/transformer.py

Line 232 in 3c1fe4e

return F.log_softmax(logit,dim=-1)

).

If we use nn.CrossEntropy for the loss function, the result of F.log_softmax enters in the loss function as in [line 333] (

PAML/model/transformer.py

Line 333 in 3c1fe4e

loss = self.criterion(logit.contiguous().view(-1, logit.size(-1)), dec_batch.contiguous().view(-1))

)

So basically the output of the projection layer goes through F.log_softmax and then nn.CrossEntropy.

However, if you look at here, simply applying nn.CrossEntropy would automatically apply F.log_softmax so I think you should exlude the line 232 and instead just return [line 218] (

PAML/model/transformer.py

Line 218 in 3c1fe4e

logit = self.proj(x)

).

What do you think ?

How you compute the second order optimization partial derivatives ?

Hi, Thanks for open-sourcing your awesome work!

I just got a little confused. In the paper, you mentioned that "This process requires second order optimization partial derivatives, which can be computed by any automatic differentiation library".

However, I did not find any code related to second order derivatives. Am I missing something? could you please kindly point out how you compute the second order derivatives?

Persona overlap between train, test and valid

Hello

Train, test and valid personas (or "tasks" ) are computed by

train = p.get_personas('train')
test = p.get_personas('test')
valid = p.get_personas('valid')

The length of test is 100 which means there are 100 distinct personas. However, 99 of them are present in train and similarly 99 of 99 personas in valid are also present in train

In addition, the difference between valid and test is only by one persona (62) so they are almost the same task

Q1.
Why every persona in test and valid are present in train ?
I thought data present in train should not appear for both test and valid

Q2.
Why do you make valid and test have almost the same personas ?

Thanks

wanting for help

today, I change a new computer to rebuild your code. But unfortunately, I met with almost the same question as before. When I am in the root directory, I run the following command(python main_fine_tune.py --cuda --model trs --batch_size 32 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 32 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/paml/${model} --save_path_dataset save/paml/ --test). Then something happened. It stopped at 49 percent of whole process,
. Then I saw the following
. So I think perhaps there is no bert-base-uncased-model there, then I downloaded it manually. So where should I put it? in the directory data\nli_model? Then I put it in the directory data\nli_model. Then the code ran pass the 49 percent, but this time at the 72 percent, the terminal prints the same information as before,
Can you help me fix this, bro?

Trained NLI model

Hello, sorry to bother you, because the Trained NLI model link you provided is no longer valid. If possible, could you please update the link? Thank you very much for your help!

expected dtype Byte but got dtype Bool

Hi！
An error occurs when I run MAML.py. Can you give me some advice? Thanks a lot!

Issue summary
Traceback (most recent call last):
File "MAML.py", line 149, in
v_loss, v_ppl = do_evaluation(meta_net, val_iter)
File "MAML.py", line 99, in do_evaluation
loss, ppl, _ = model.train_one_batch(batch, train=False)
File "C:\Users\lenovo\Desktop\PAML\PAML\model\transformer.py", line 329, in train_one_batch
pre_logit, attn_dist = self.decoder(self.embedding(dec_batch_shift),encoder_outputs, (mask_src,mask_trg))
File "C:\Users\lenovo\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "C:\Users\lenovo\Desktop\PAML\PAML\model\transformer.py", line 177, in forward
dec_mask = torch.gt(mask_trg + self.mask[:, :mask_trg.size(-1), :mask_trg.size(-1)], 0)

RuntimeError: expected device cuda:0 and dtype Byte but got device cuda:0 and dtype Bool

[Data Issue]

Help needed，THX！

When I finished training paml, then I want to fine-tine and test paml. But I found there is no trained model in the path ./save/paml. Nevertheless，there are indeed models in the ./save/no_persona and ./save/persona. How can I fix this problem? Guys, I need your help!

I want to know the usage of persona_map

What is the purpose of data_analysis.ipynb？Is it necessary for the code?Is there another way to get persona_map file?

Error occurs in out-loop backward

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

I think the backward operation in train_one_batch (line 308 in transformer.py) has freed the graph, while this does not disturb the out-loop backward.

The error occurs when batch_loss.backward() (line 167 in MAML.py)

Error about meta_optimizer and new data

Hello! I use my own data to train your model. After line 170 meta_optimizer.step() is executed, line 150 val_loss, v_ppl = do_learning_fix_step(meta_net, train_iter, val_iter, iterations=config.meta_iteration) outputs val_loss as tensor(nan, device='cuda:0', grad_fn=<AddBackward>), which causes the training failure. I didn't change your code but persona_map, and I want to know what went wrong. Thanks!

k-shot consistency C score is different from what mentioned in paper

First, I would like to thanks for your contribution.
I trained your model exactly like what you said in document.But I got Entl_b = 0.0879 in printed result , I checked and found out that is the C score ( am i right ? ) ! The problem is that in paper C score has been reported equal to 0.2 .
By the way the Entl_b that I mention before was for checkpoint with loss of 46.5833 ( last checkpoint )
Thanks in advance

Some doubts about the dataset

I'd like to use the same dataset to train another persona chatbot model, but I have no idea what the cand mean in the train set. What's the association between cand and u/r ?

Question about dataset

Hello
I downloaded the ConvAI2 dataset from ParlAI, and the size of train_self_original.txt was something around two times bigger than what is inside the repository. I would like to ask, why the size is different? And which dataset should be used for training purposes.
And I would like to know how *_persona_map files were created?

Thanks in advance

Codes to produce the paml_generation file

Hi, thanks for providing the paml_generation file which stores a lot of samples. Are there any codes or functions for printing the samples in the generation phase? Or any codes or functions to produce the paml_generation file?

Hi, sorry to bother you. I have some questions to ask you for help.

When I train your PAML model, I run this command at my terminal,
( python main.py --cuda --model trs --pretrain_emb --weight_sharing --label_smoothing --noam --emb_dim 300 --hidden_dim 300 --pointer_gen --save_path save/no_persona/)
then the terminal showed that this error,

then I check your repo for the same path, there is truly no this file. I am sincerely asking for your help.

Problem while loading BERT pre-trained model for the NLI-Scorer

sorry to bother you. I need your help. when I run the command(python MAML.py --cuda --model trs --batch_size 16 --use_sgd --lr 0.01 --meta_lr 0.0003 --meta_batch_size 16 --meta_optimizer adam --pretrain_emb --weight_sharing --emb_dim 300 --hidden_dim 300 --fix_dialnum_train --pointer_gen --save_path save/paml/)
I got the following result

so, can you give me some advice on fixing the problem? thx.