ashoknp-git/awesome-sentence-embedding

A curated list of pretrained sentence and word embedding models

awesome-sentence-embedding

A curated list of pretrained sentence and word embedding models

Table of Contents

• About This Repo
• General Framework
• Word Embeddings
• OOV Handling
• Contextualized Word Embeddings
• Pooling Methods
• Encoders
• Evaluation
• Misc
• Vector Mapping
• Articles

About This Repo

• well there are some awesome-lists for word embeddings and sentence embeddings, but all of them are outdated and more importantly incomplete
• this repo will also be incomplete, but I'll try my best to find and include all the papers with pretrained models
• this is not a typical awesome list because it has tables but I guess it's ok and much better than just a huge list
• if you find any mistakes or find another paper or anything please send a pull request and help me to keep this list up to date
• enjoy!

General Framework

• Almost all the sentence embeddings work like this:
• Given some sort of word embeddings and an optional encoder (for example an LSTM) they obtain the contextualized word embeddings.
• Then they define some sort of pooling (it can be as simple as last pooling).
• Based on that they either use it directly for the supervised classification task (like infersent) or generate the target sequence (like skip-thought).
• So, in general, we have many sentence embeddings that you have never heard of, you can simply do mean-pooling over any word embedding and it's a sentence embedding!

Word Embeddings

• Note: don't worry about the language of the code, you can almost always (except for the subword models) just use the pretrained embedding table in the framework of your choice and ignore the training code

date	paper	citation count	training code	pretrained models
-	WebVectors: A Toolkit for Building Web Interfaces for Vector Semantic Models	N/A	-	RusVectōrēs(broken)
2013/01	Efficient Estimation of Word Representations in Vector Space	999+	C	Word2Vec
2014/12	Word Representations via Gaussian Embedding	123	Cython	-
2014/??	Dependency-Based Word Embeddings	510	C++	word2vecf
2014/??	A Probabilistic Model for Learning Multi-Prototype Word Embeddings	80	DMTK	-
2014/??	GloVe: Global Vectors for Word Representation	999+	C	GloVe
2015/06	From Paraphrase Database to Compositional Paraphrase Model and Back	0	Theano	PARAGRAM
2015/06	Non-distributional Word Vector Representations	37	Python	WordFeat
2015/06	Sparse Overcomplete Word Vector Representations	68	C++	-
2015/??	Topical Word Embeddings	131	Cython
2015/??	SensEmbed: Learning Sense Embeddings for Word and Relational Similarity	138	-	SensEmbed
2015/??	Joint Learning of Character and Word Embeddings	92	C	-
2016/02	Swivel: Improving Embeddings by Noticing What's Missing	34	TF	-
2016/03	Counter-fitting Word Vectors to Linguistic Constraints	103	Python	counter-fitting(broken)
2016/05	Mixing Dirichlet Topic Models and Word Embeddings to Make lda2vec	24	Chainer	-
2016/06	Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations	30	Go	lexvec
2016/06	Siamese CBOW: Optimizing Word Embeddings for Sentence Representations	90	Theano	Siamese CBOW
2016/07	Enriching Word Vectors with Subword Information	999+	C++	fastText
2016/08	Morphological Priors for Probabilistic Neural Word Embeddings	18	Theano	-
2016/11	A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks	144	C++	charNgram2vec
2016/12	ConceptNet 5.5: An Open Multilingual Graph of General Knowledge	122	Python	Numberbatch
2016/??	Learning Word Meta-Embeddings	19	-	Meta-Emb(broken)
2017/02	Offline bilingual word vectors, orthogonal transformations and the inverted softmax	127	Python	-
2017/04	Multimodal Word Distributions	27	TF	word2gm
2017/05	Poincaré Embeddings for Learning Hierarchical Representations	106	Pytorch	-
2017/06	Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints	0	TF	Attract-Repel
2017/06	Context encoders as a simple but powerful extension of word2vec	1	Python	-
2017/08	Learning Chinese Word Representations From Glyphs Of Characters	10	C	-
2017/08	Making Sense of Word Embeddings	15	Python	sensegram
2017/09	Hash Embeddings for Efficient Word Representations	5	Keras	-
2017/10	BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages	11	Gensim	BPEmb
2017/11	SPINE: SParse Interpretable Neural Embeddings	8	Pytorch	SPINE
2017/??	Dict2vec : Learning Word Embeddings using Lexical Dictionaries	14	C++	Dict2vec
2017/??	Joint Embeddings of Chinese Words, Characters, and Fine-grained Subcharacter Components	14	C	-
2017/??	AraVec: A set of Arabic Word Embedding Models for use in Arabic NLP	21	Gensim	AraVec
2017/??	Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics	8	C	-
2018/04	Representation Tradeoffs for Hyperbolic Embeddings	18	Pytorch	h-MDS
2018/04	Dynamic Meta-Embeddings for Improved Sentence Representations	6	Pytorch	DME/CDME
2018/05	Analogical Reasoning on Chinese Morphological and Semantic Relations	15	-	ChineseWordVectors
2018/06	Probabilistic FastText for Multi-Sense Word Embeddings	8	C++	Probabilistic FastText
2018/09	Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks	1	TF	SynGCN
2018/09	FRAGE: Frequency-Agnostic Word Representation	6	Pytorch	-
2018/12	Wikipedia2Vec: An Optimized Tool for LearningEmbeddings of Words and Entities from Wikipedia	0	Cython	Wikipedia2Vec
2018/??	Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings	5	-	ChineseEmbedding
2018/??	cw2vec: Learning Chinese Word Embeddings with Stroke n-gram Information	7	C++	-
2019/02	VCWE: Visual Character-Enhanced Word Embeddings	0	Pytorch	VCWE
2019/05	Learning Cross-lingual Embeddings from Twitter via Distant Supervision	0	Text	-

OOV Handling

• Drop OOV words!
• One OOV vector(unk vector)
• Use subword models(ngram, bpe, char)
• ALaCarte: A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors
• Mimick: Mimicking Word Embeddings using Subword RNNs
• CompactReconstruction: Subword-based Compact Reconstruction of Word Embeddings

Contextualized Word Embeddings

• Note: all the unofficial models can load the official pretrained models

date	paper	citation count	code	pretrained models
-	Language Models are Unsupervised Multitask Learners	N/A	TF Pytorch Keras	GPT-2
2017/08	Learned in Translation: Contextualized Word Vectors	175	Pytorch Keras	CoVe
2018/01	Universal Language Model Fine-tuning for Text Classification	176	Pytorch	ULMFit(English, Zoo)
2018/02	Deep contextualized word representations	674	Pytorch TF	ELMO(AllenNLP, TF-Hub)
2018/04	Efficient Contextualized Representation:Language Model Pruning for Sequence Labeling	3	Pytorch	LD-Net
2018/07	Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation	10	Pytorch	ELMo
2018/08	Direct Output Connection for a High-Rank Language Model	5	Pytorch	DOC
2018/10	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	507	TF Keras Pytorch MXNet PaddlePaddle TF Keras	BERT(BERT, ERNIE)
2018/??	Contextual String Embeddings for Sequence Labeling	13	Pytorch	Flair
2018/??	Improving Language Understanding by Generative Pre-Training	180	TF Keras Pytorch	GPT
2019/01	Cross-lingual Language Model Pretraining	20	Pytorch	XLM
2019/01	Multi-Task Deep Neural Networks for Natural Language Understanding	22	Pytorch	MT-DNN
2019/01	Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context	23	TF Pytorch Pytorch	Transformer-XL
2019/01	BioBERT: pre-trained biomedical language representation model for biomedical text mining	7	TF	BioBERT
2019/03	SciBERT: Pretrained Contextualized Embeddings for Scientific Text	3	Pytorch, TF	SciBERT
2019/04	ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission	2	Pytorch	ClinicalBERT
2019/04	Publicly Available Clinical BERT Embeddings	3	Text	clinicalBERT
2019/05	ERNIE: Enhanced Language Representation with Informative Entities	0	Pytorch	ERNIE
2019/06	Pre-Training with Whole Word Masking for Chinese BERT	N/A	Pytorch, TF	BERT-wwm
2019/06	XLNet: Generalized Autoregressive Pretraining for Language Understanding	N/A	TF	XLNet

Pooling Methods

• {Last, Mean, Max}-Pooling
• Special Token Pooling (like BERT and OpenAI's Transformer)
• SIF: A Simple but Tough-to-Beat Baseline for Sentence Embeddings
• TF-IDF: Unsupervised Sentence Representations as Word Information Series: Revisiting TF--IDF
• P-norm: Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
• DisC: A Compressed Sensing View of Unsupervised Text Embeddings, Bag-of-n-Grams, and LSTMs
• GEM: Zero-Training Sentence Embedding via Orthogonal Basis
• SWEM: Baseline Needs More Love: On Simple Word-Embedding-Based Modelsand Associated Pooling Mechanisms
• VLAWE: Vector of Locally-Aggregated Word Embeddings (VLAWE): A Novel Document-level Representation

Encoders

date	paper	citation count	code	model_name
2014/05	Distributed Representations of Sentences and Documents	999+	Pytorch Python	Doc2Vec
2014/11	Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models	437	Theano Pytorch	VSE
2015/06	Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books	218	Theano TF Pytorch, Torch	SkipThought
2015/11	Order-Embeddings of Images and Language	165	Theano	order-embedding
2015/11	Towards Universal Paraphrastic Sentence Embeddings	218	Theano	ParagramPhrase
2015/??	From Word Embeddings to Document Distances	412	C, Python	Word Mover's Distance
2016/02	Learning Distributed Representations of Sentences from Unlabelled Data	212	Python	FastSent
2016/07	Charagram: Embedding Words and Sentences via Character n-grams	70	Theano	Charagram
2016/11	Learning Generic Sentence Representations Using Convolutional Neural Networks	34	Theano	ConvSent
2017/03	Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features	105	C++	Sent2Vec
2017/04	Learning to Generate Reviews and Discovering Sentiment	132	TF Pytorch Pytorch	Sentiment Neuron
2017/05	Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings	33	Theano	GRAN
2017/05	Supervised Learning of Universal Sentence Representations from Natural Language Inference Data	341	Pytorch	InferSent
2017/07	VSE++: Improving Visual-Semantic Embeddings with Hard Negatives	74	Pytorch	VSE++
2017/08	Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm	N/A	Keras Pytorch	DeepMoji
2017/09	StarSpace: Embed All The Things!	40	C++	StarSpace
2017/10	DisSent: Learning Sentence Representations from Explicit Discourse Relations	31	Pytorch	DisSent
2017/11	Dual-Path Convolutional Image-Text Embedding with Instance Loss	1	Matlab	Image-Text-Embedding
2017/11	Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations	31	Theano	para-nmt
2018/03	An efficient framework for learning sentence representations	44	TF	Quick-Thought
2018/03	Universal Sentence Encoder	69	TF-Hub	USE
2018/04	End-Task Oriented Textual Entailment via Deep Explorations of Inter-Sentence Interactions	2	Theano	DEISTE
2018/04	Learning general purpose distributed sentence representations via large scale multi-task learning	65	Pytorch	GenSen
2018/06	Embedding Text in Hyperbolic Spaces	14	TF	HyperText
2018/07	Representation Learning with Contrastive Predictive Coding	51	Keras	CPC
2018/08	Context Mover’s Distance & Barycenters: Optimal transport of contexts for building representations	0	Python	CMD
2018/09	Learning Universal Sentence Representations with Mean-Max Attention Autoencoder	2	TF	Mean-MaxAAE
2018/10	Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model	2	TF-Hub	USE-xling
2018/10	BioSentVec: creating sentence embeddings for biomedical texts	3	Python	BioSentVec
2018/11	Word Mover's Embedding: From Word2Vec to Document Embedding	6	C, Python	WordMoversEmbeddings
2018/11	A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks	9	Pytorch	HMTL
2018/12	Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond	11	Pytorch	LASER
2018/??	Convolutional Neural Network for Universal Sentence Embeddings	0	Theano	CSE
2019/01	No Training Required: Exploring Random Encoders for Sentence Classification	7	Pytorch	randsent
2019/02	CBOW Is Not All You Need: Combining CBOW with the Compositional Matrix Space Model	0	Pytorch	CMOW

Evaluation

• decaNLP: The Natural Language Decathlon: Multitask Learning as Question Answering
• SentEval: SentEval: An Evaluation Toolkit for Universal Sentence Representations
• GLUE: GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Exploring Semantic Properties of Sentence Embeddings
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Word Embeddings Benchmarks: How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks
• MLDoc: A Corpus for Multilingual Document Classification in Eight Languages
• LexNET: Olive Oil Is Made of Olives, Baby Oil Is Made for Babies: Interpreting Noun Compounds Using Paraphrases in a Neural Model
• wordvectors.net: Community Evaluation and Exchange of Word Vectors at wordvectors.org
• jiant: Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling
• jiant: What do you learn from context? Probing for sentence structure in contextualized word representations
• Evaluation of sentence embeddings in downstream and linguistic probing tasks
• QVEC: Evaluation of Word Vector Representations by Subspace Alignment
• Grammatical Analysis of Pretrained Sentence Encoders with Acceptability Judgments
• EQUATE : A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference
• Evaluating Word Embedding Models: Methods andExperimental Results
• How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions
• Linguistic Knowledge and Transferability of Contextual Representations: contextual-repr-analysis
• LINSPECTOR: Multilingual Probing Tasks for Word Representations

Misc

• Word Embedding Dimensionality Selection: On the Dimensionality of Word Embedding
• Half-Size: Simple and Effective Dimensionality Reduction forWord Embeddings
• magnitude: Magnitude: A Fast, Efficient Universal Vector Embedding Utility Package
• To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
• Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors: fuzzymax
• The Pupil Has Become the Master: Teacher-Student Model-BasedWord Embedding Distillation with Ensemble Learning: EmbeddingDistillation
• Improving Distributional Similarity with Lessons Learned from Word Embeddings: hyperwords

Vector Mapping

• Cross-lingual Word Vectors Projection Using CCA: Improving Vector Space Word Representations Using Multilingual Correlation
• vecmap: A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings
• MUSE: Unsupervised Machine Translation Using Monolingual Corpora Only
• CrossLingualELMo: Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing

Articles

• Comparing Sentence Similarity Methods
• The Current Best of Universal Word Embeddings and Sentence Embeddings
• On sentence representations, pt. 1: what can you fit into a single #$!%@*&% blog post?
• Deep-learning-free Text and Sentence Embedding, Part 1
• Deep-learning-free Text and Sentence Embedding, Part 2
• An Overview of Sentence Embedding Methods
• Word embeddings in 2017: Trends and future directions
• A Walkthrough of InferSent – Supervised Learning of Sentence Embeddings
• A survey of cross-lingual word embedding models
• Introducing state of the art text classification with universal language models