zminghua/sentencoding

Sentence encoder and training code for Mean-Max AAE

Mean-Max AAE

Sentence encoder and training code for the paper Learning Universal Sentence Representations with Mean-Max Attention Autoencoder.

Dependencies

This code is written in python. To use it you will need:

• Python 2.7
• TensorFlow
• NumPy
• NLTK
• SciPy
• Ray (for parallel evaluation on transfer tasks)

Download datasets

The pre-processed Toronto BookCorpus we used for training our model is available here.

To download GloVe vector:

curl -Lo data/glove.840B.300d.zip http://nlp.stanford.edu/data/glove.840B.300d.zip
unzip data/glove.840B.300d.zip -d data/

To get all the transfer tasks datasets, run (in data/):

./get_transfer_data.bash

This will automatically download and preprocess the transfer tasks datasets, and store them in data/.

Sentence encoder

We provide a simple interface to encode English sentences. Get started with the following steps:

1) Download our Mean-Max AAE models and put it to SentEncoding directory, then decompress:

unzip models.zip

2) Make sure you have the NLTK tokenizer by running the following once:

import nltk
nltk.download('punkt')

3) Load our pre-trained model:

import master
m = master.Master('conf.json')
m.creat_graph()
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
m.prepare()

3) Build the vocabulary of word vectors (i.e keep only those needed):

vocab = m.build_vocab(sentences, tokenize=True)
m.build_emb(vocab)

where sentences is your list of n sentences.

4) Encode your sentences:

embeddings = m.encode(sentences, tokenize=True)

This outputs a numpy array with n vectors of dimension 4096.

Reference

If you found this code useful, please cite the following paper:

  @inproceedings{zhang2018learning,
  title={Learning Universal Sentence Representations with Mean-Max Attention Autoencoder},
  author={Zhang, Minghua and Wu, Yunfang and Li, Weikang and Li, Wei},
  booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  pages={4514--4523},
  year={2018}
  }

sentencoding's Issues

'float' object cannot be interpreted as an integer

When execute the encode function, the following error occurs:

---

TypeError Traceback (most recent call last)
in
----> 1 m.encode(['123','456'], tokenize=True)

/apps/dataiku-data/jupyter-run/dku-workdirs/MARRIOTT/SentEncoding/master.py in encode(self, task_data, tokenize, use_norm)
261 for l in ds.keys():
262 numbatches = len(ds[l]) / self.conf['option']['encode_bs'] + 1
--> 263 for minibatch in range(numbatches):
264 caps = ds[l][minibatch::numbatches]
265

TypeError: 'float' object cannot be interpreted as an integer

Solved:
Cannot encode words outside dictionary.

BookCorpus dataset not available

Hello,

I went to the link http://yknzhu.wixsite.com/mbweb, however the BookCorpus dataset is no longer available. Could you let me know what other dataset we can use for training as it's showing me this error when I run run_train.py?

FileNotFoundError: [Errno 2] No such file or directory: '/content/SentEncoding/data/bookcorpus/books.vocab'

Thanks.

where is the code ?