ssemeniuta/textvae

Theano code for experiments in the paper "A Hybrid Convolutional Variational Autoencoder for Text Generation."

A Hybrid Convolutional Variational Autoencoder for Text Generation.

Theano code for experiments in the paper A Hybrid Convolutional Variational Autoencoder for Text Generation.

Preparation

First, run makedata.sh. This will download the ptb dataset, split, and preprocess it.

PTB Experiments

Files prefixed with ''lm_'' contain experiments on the ptb dataset. We provide scripts for training of non-VAE, baseline LSTM VAE, and our models and a script to greedily sample from a trained model. ''defs'' subfolder contains definitions of grid searches we have used to generate data for figures and tables in the paper. Running one search is done by:

python -u nn/scripts/grid_search.py -grid defs/gridname.json

To train our model on samples 60 characters long with alpha=0.2 run:

python -u lm_vae_lstm.py -alpha 0.2 -sample_size 60

Twitter Experiments

Code for these experiments is in files starting with ''twitter_''. We do not release the dataset we have used to train our model, but provide both a script to train one and a pretrained model. To use the script on custom data, create a file ''data/tweets.txt'' containing one data sample per line. By default, the first 10k samples will be used for validation and everything else for training, but no more than ~1M samples. In addition, it will only use tweets with up to 128 characters. This is done only for convenience when down- and upsampling. Training on tweets with up to 140 characters will require a little bit of care when handling spatial dimension.

License

MIT

textvae's Issues

Negative KL divergence?

Hello! I have a simple and possibly silly question here. In my training runs, the kld term, as logged to costs by vae.py, is always negative, creeping upwards toward zero. Is that expected behavior?

Then, I see that the kld is subtracted from reconstruction_loss in the calculation of the overall cost. Shouldn't kld be positive, and added to reconstruction_loss?

Here's an example of what I'm seeing:

(I've set anneal_start=0 and anneal_end=100 here just for demonstration purposes.)

Thanks for a great paper and a very usable code release!

LICENSE?

What is the license for the Theano code in the repo?

Broken link to pretrained twitter model

Dropbox link is no longer working :)

ValueError: cannot reshape array of size 100 into shape (259,)

loading exp/twittervae.charlevel.z_100.len_128.p_0.00.lstmsz_1000.alpha_0.20/model.flt
File "C:/Users/eyaler/Dropbox/python/text_vae/textvae/twitter_vae_sample_charlevel.py", line 163, in
main(**vars(args))
File "C:/Users/eyaler/Dropbox/python/text_vae/textvae/twitter_vae_sample_charlevel.py", line 31, in main
model.load("exp/%s/model.flt" % name)
File "C:\Users\eyaler\Dropbox\python\text_vae\textvae\nn\models\base_model.py", line 89, in load
params = flt.load()
File "C:\Users\eyaler\Dropbox\python\text_vae\textvae\nn\utils.py", line 77, in load
arr = self.load_array()
File "C:\Users\eyaler\Dropbox\python\text_vae\textvae\nn\utils.py", line 104, in load_array
return arr.reshape(dims)
ValueError: cannot reshape array of size 100 into shape (259,)