snakeztc/neuraldialog-cvae

Tensorflow Implementation of Knowledge-Guided CVAE for dialog generation ACL 2017. It is released by Tiancheng Zhao (Tony) from Dialog Research Center, LTI, CMU

Knowledge-Guided CVAE for dialog generation

We provide a TensorFlow implementation of the CVAE-based dialog model described in Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders, published as a long paper in ACL 2017. See the paper for more details.

References

If you use any source codes or datasets included in this toolkit in your work, please cite the following paper. The bibtex are listed below:

[Zhao et al, 2017]:
 @inproceedings{zhao2017learning,
   title={Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders},
   author={Zhao, Tiancheng and Zhao, Ran and Eskenazi, Maxine},
   booktitle={Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
   volume={1},
   pages={654--664},
   year={2017}
 }

External Implementation

The baseline method HRED using the same SwitchBoard dataset is also implemented on Texar, a general-purpose text generation toolkit. Checkout here.

Prerequisites

• TensorFlow 1.3.0
• cuDNN 6
• Python 2.7
• Numpy
• NLTK
• You may need to pip install beeprint if the module is missing

Usage

Train a new model

python kgcvae_swda.py

will run default training and save model to ./working

Test a existing model

Modify the TF flags at the top of kgcvae_swda.py as follows to run a existing model

forward_only: False -> True
test_path: set to the folder contains the model. E.g. runxxxx

Then you can run the model by:

python kgcvae_swda.py

The outputs will be printed to stdout and generated responses will be saved at test.txt in the test_path.

Use pre-trained Word2vec

Download Glove word embeddings from https://nlp.stanford.edu/projects/glove/ The default setting use 200 dimension word embedding trained on Twitter.

At last, set word2vec_path at line 15 of kgcvae_swda.py.

Dataset

We release two dataset:

1. full_swda_clean_42da_sentiment_dialog_corpus.p is a binary dump using python Pickle library that contains the raw data and used for training
2. json_format: the same dialog data also is presented in JSONL format in the data directory.
3. test_mutl_ref.json is only the test data set with multiple references responses with dialog act annotations. The multiple referneces are collected according to the method described in the Appendix of the paper.

Data Format

If you want to train the model on your own data. Please create a pickle file has the following format:

# The top directory is a python dictionary
type(data) = dict
data.keys() = ['train', 'valid', 'test']

# Train/valid/test is a list, each element is one dialog
train = data['train']
type(train) = list

# Each dialog is a dict
dialog = train[0]
type(dialog)= dict
dialog.keys() = ['A', 'B', 'topic', 'utts']

# A, B contain meta info about speaker A and B.
# topic defines the dialog prompt topic in Switchboard Corpus.

# utts is a list, each element is a tuple that contain info about an utterance
utts = dialog['utts']
type(utts) = list
utts[0] = ("A" or "B", "utterance in string", [dialog_act, other_meta_info])

# For example, a utterance look like this:
('B','especially your foreign cars',['statement-non-opinion'])

Put the resulting file into ./data and set the data_dir in kgcvae_swda.py

neuraldialog-cvae's Issues

Error when trying to train num_layer > 1

Hi,

When I set:

cxt_cell_size = 150  # context encoder hidden size
sent_cell_size = 75  # utterance encoder hidden size
dec_cell_size = 100  # response decoder hidden size
num_layer = 4  # number of context RNN layers

I get the following error:

ValueError: Trying to share variable model/contextRNN/rnn/multi_rnn_cell/cell_0/gru_cell/gates/kernel, but specified shape (300, 300) and found shape (302, 300).

Confusing about the evaluation

Hi,

I recently running your code, and get this after training with glove, I find it is confusing that the evaluation has no direct connection with results in your paper. Could you give a more detailed explain or more consist evaluation?

What modifications needed to train dataset without dialogue acts, metadata and features and floor?

Hi,

What modifications in the code are needed to train on a dataset without dialogue acts, metadata, and features?

ValueError: num_outputs should be int or long, got 400.

Traceback (most recent call last):
File "D:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 2862, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
runfile('G:/Users/PycharmProjects/untitl2/NeuralDialog-CVAE-master/NeuralDialog-CVAE-master/kgcvae_swda.py', wdir='G:/Users/PycharmProjects/untitl2/NeuralDialog-CVAE-master/NeuralDialog-CVAE-master')
File "G:\Program Files\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "G:\Program Files\JetBrains\PyCharm 2020.1\plugins\python\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "G:/Users/PycharmProjects/untitl2/NeuralDialog-CVAE-master/NeuralDialog-CVAE-master/kgcvae_swda.py", line 169, in
main()
File "G:/Users/PycharmProjects/untitl2/NeuralDialog-CVAE-master/NeuralDialog-CVAE-master/kgcvae_swda.py", line 72, in main
model = KgRnnCVAE(sess, config, api, log_dir=None if FLAGS.forward_only else log_dir, forward=False, scope=scope)
File "G:\Users\PycharmProjects\untitl2\NeuralDialog-CVAE-master\NeuralDialog-CVAE-master\models\cvae.py", line 257, in init
activation_fn=tf.tanh, scope="fc1")
File "D:\ProgramData\Anaconda3\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 181, in func_with_args
return func(*args, **current_args)
File "D:\ProgramData\Anaconda3\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1638, in fully_connected
'num_outputs should be int or long, got %s.' % (num_outputs,))
ValueError: num_outputs should be int or long, got 400.

how to solve it ?

Evaluation

Hi, could you provide the evaluation code to reproduce the results of the paper?

about performance

I run your code with python kgcvae_swda.py on cpu, and get the output:

Avg recall BLEU 0.358250, avg precision BLEU 0.124279 and F1 0.184540 (only 1 reference response. Not final result)
Avg recall BLEU 0.358250, avg precision BLEU 0.124279 and F1 0.184540 (only 1 reference response. Not final result)

I used python3.6 and tensorflow 1.12 cpu version, and made neccessary changes to run your code in my environment. But it seems something went wrong.

Also I ran your code on python2 and tensorflow 1.3 cpu version, but got the same result. Can you give some tips to reproduce your results? Thanks.

LSTM doesn't work

Changing the cell type to LSTM throws an error (with default parameters and code):

ValueError: Shape must be rank 2 but is rank 3 for 'model/concat' (op: 'ConcatV2') with input shapes: [?,30], [?,4], [?,4], [2,?,600], [].

Glove word embeddings assignment

If Glove word embeddings is assigned before tf.global_variables_initializer(). The embedding will be randomly initialized. It will not use Glove word embeddings.

Can we use your data in PaddlePaddle NLP's VAE example ?

Hello, I am a member of the PaddlePaddle NLP team. Can we use your data in our VAE example ? We will note the source of your data in the README file. Thank you so much.

About printing dialog act in test batch.

NeuralDialog-CVAE/models/cvae.py

Line 535 in 6376dbe

pred_da = np.argmax(sample_das[r_id], axis=1)[0]

Dear Dr. Zhao. I'm Sangjun Koo from POSTECH
(You may have heard of POSTECH from Dr. Kyusong Lee, one of my former advisor.)

Thanks for releasing your code for further research.
Would you mind me asking some question about your code?

In reimplementing your code into PyTorch, I found that some anomaly which could be a bug(although it is not serious). The line should be np.argmax(sample_das[r_id], axis=1)[bid] instead of np.argmax(sample_das[r_id], axis=1)[0] since this may cause to print only the first predicted DA element in the batch.

Any comments would be welcome.

questions about the sampling strategy for baseline model

in your paper, you have said: 【Also, to compare the diversity introduced by the stochasticity in the proposed latent variable versus the softmax of RNN at each decoding step, we generate N responses from the baseline by sampling from the softmax. For CVAE/kgCVAE, we sample N times from the latent z and only use greedy decoders so that the randomness comes entirely from the latent variable z.】

the tradictional beam search with size B have two step, first for each beam, generate top-B words from the vocab-softmax,
then generate top-B beams from the B*B candidate sequences using the average probability.
Is the sampling in the above figure means two multinomial step for the inner vocabulary softmax and the outer average probability?
And is the inner sampling with replacement or without replacement? Is the outer sampling with replacement or without replacement?

Problem about the multiple references test dataset

Hi,

I want to use the test dataset with multiple references(which you uploaded at https://github.com/snakeztc/NeuralDialog-CVAE/blob/master/data/test_multi_ref.json.) to get the evaluation result you presented in the paper.
But I'm pretty confused because the format of this one is quite different from the one with single reference, for it's in lack of some information, such as topic.
Could you please tell me how to get the evaluation result with multiple references with this test dataset which seems to be incomplete?

Looking forward to your reply. ：）

import tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl as rnn_cell ImportError: No module named core_rnn_cell_impl

Tensorflow 0.12.1

Traceback (most recent call last):
File "kgcvae_swda.py", line 13, in
from models.cvae import KgRnnCVAE
File "/home/ubuntu/NeuralDialog-CVAE/models/cvae.py", line 8, in
import tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl as rnn_cell
ImportError: No module named core_rnn_cell_impl

Which tensorflow version works fine with it?

The same question about Evaluation

Hi, could you provide the evaluation code to reproduce the results of the paper?

bow loss

Why do you calculate the the bow loss similar to a sequence (tile bow logits to max_out_len and compare it to labels) instead of forming a single one hot vector out of the labels and compare it to the predicted bow vector?

Diversity

There is no content on diversity comparison in the paper. When I used the CVAE model to build my system, I compared it with the Seq2Seq structure and found that the performance of Seq2Seq was greatly influenced by the P-value of Top-P sampling, while the performance of CVAE structure was greatly affected by the weight ratio of reconstruction loss and KL loss. I wonder if you considered the issue of diversity when conducting the experiment？

look forward to your reply～

How can I ignore <unk> tokens while decoding?

Hi, how can I ignore unk tokens while decoding? It probably just means setting the weight to zero.

So I want to do something like this:

if ignore_unk:
    next_probs[:, self.unk_id] = 0

How can I do this in your code?

Thanks.