derooce/awesome-pretrained-language-model

Worth-reading papers and related resources on pretrained-language models(PLMs). On the Shoulder of Giants!

Awesome Pretrained Language Models

Pretrained Language Models (PLMs) have been achieved great success over many NLP tasks. And with the rapid development of PLMs, there is a need to make a list to show these succesful and remarkable language models.

Lists of Language Models

Group by companies and organizations.

• Awesome Pretrained Language Models
• Lists of Language Models
  • Al21 Labs
  • Alibaba
  • Amazon
  • Baidu
  • BigSicence
  • DeepMind
  • EleutherAI
  • Meta (Facebook)
  • Fudan University
  • Google
  • HIT & iFLYTEK
  • Huawei
  • Inspur
  • IDEA
  • JD (JINGDONG)
  • Langboat
  • Microsoft
  • Nvidia
  • OpenAI
  • OPPO
  • Tencent
  • THU (Tsinghua) & BAAI (Beijing)
  • More PLMs are coming ...
• Claims

Al21 Labs

• Jurassic-1
  • JURASSIC-1: TECHNICAL DETAILS AND EVALUATION
  • github repo

Alibaba

• PLUG (Pre-training for Language Understanding and Generation))
  • 超大规模中文生成

Amazon

• Alexa Teacher Model
  • Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems
  • AlexaTM 20B
    • AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model
  • github repo (model hasn't been publicly released yet, stay tuned!)

Baidu

• ERNIE family: github repo
  • ERNIE 1.0
    • ERNIE: Enhanced Representation through Knowledge Integration
  • ERNIE 2.0
    • ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
  • ERNIE 3.0
    • ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
  • ERNIE 3.0 TiTan
    • ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
  • ERNIE 3.0 Zeus
    • ERNIE 3.0 Zeus Prompt: 全球首个融合任务相关知识的千亿中文大模型
  • ERNIE-GEN
    • ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation
  • ERNIE-ViL
    • ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
  • ERNIE-ViLG
    • ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation
  • ERNIE-Gram
    • ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding
  • ERNIE-M
    • ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora
  • ERNIE-Doc
    • ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
  • ERNIE-Tiny
    • ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
  • ERNIE-GeoL
    • ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
  • ERNIE-SPARSE
    • ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention
  • ERNIE-Search
    • ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval

• PLATO family: github repo
  • PLATO-1
    • PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable
  • PLATO-2
    • PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learning
  • PLATO-XL
    • PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
  • PLATO-KAG
    • PLATO-KAG: Unsupervised Knowledge-Grounded Conversation via Joint Modeling

BigSicence

• T0 series
  • Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Hugging Face 🤗
  • github repo
• BLOOM
  • Hugging Face 🤗

DeepMind

• Gopher
  • Scaling Language Models: Methods, Analysis & Insights from Training Gopher
• Chinchilla
  • Training Compute-Optimal Large Language Models
• Retro
  • Improving language models by retrieving from trillions of tokens
  • github repo (Non-offical implementation)

EleutherAI

Hugging Face 🤗

• GPT-Neo
  • github repo
• GPT-J
  • github repo
• GPT-NeoX
  • github repo

Meta (Facebook)

• XGLM
  • Few-shot Learning with Multilingual Language Models
  • Hugging Face 🤗
• XLM-R
  • Unsupervised Cross-lingual Representation Learning at Scale
  • github repo
  • Hugging Face 🤗
• M2M-100
  • Beyond English-Centric Multilingual Machine Translation
  • github page
  • Hugging Face 🤗
• Blender
  • Recipes for building an open-domain chatbot
  • Internet-Augmented Dialogue Generation
  • Beyond Goldfish Memory: Long-Term Open-Domain Conversation
  • BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
  • github repo
  • BlenderBot A CONVERSATIONAL AI PROTOTYPE
  • Hugging Face 🤗
• OPT
  • OPT: Open Pre-trained Transformer Language Models
  • github repo
  • Hugging Face 🤗
  • Request access to OPT-175B
• NLLB
  • No Language Left Behind: Scaling Human-Centered Machine Translation
  • github repo
  • Hugging Face 🤗
• Altas
  • Few-shot Learning with Retrieval Augmented Language Models
• PEER
  • PEER: A Collaborative Language Model

Fudan University

• CPT
  • CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
  • github repo

Google

• T5 family
  • T5 series
    • Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
    • GLU Variants Improve Transformer
    • github repo
    • Hugging Face 🤗
  • ByT5
    • ByT5: Towards a token-free future with pre-trained byte-to-byte models
    • github repo
    • Hugging Face 🤗
  • ExT5
    • ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
  • LongT5
    • LongT5: Efficient Text-To-Text Transformer for Long Sequences
    • github repo
    • Hugging Face 🤗
• LaMDA
  • LaMDA: Language Models for Dialog Applications
  • github repo (Non-offical)
• FLAN
  • Finetuned Language Models Are Zero-Shot Learners
  • github repo
• GShard
  • GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Meena
  • Towards a Human-like Open-Domain Chatbot
  • github repo (Non-offical)
• PaLM
  • PaLM: Scaling Language Modeling with Pathways
  • github repo (Non-offical)
• UL2
  • Unifying Language Learning Paradigms
  • github repo
  • Hugging Face 🤗
• Minerva
  • Solving Quantitative Reasoning Problems with Language Models
  • Minerva SAMPLE EXPLORER
• Flan-PaLM
  • Scaling Instruction-Finetuned Language Models
  • Flan-T5 checkpoints

HIT & iFLYTEK

• PERT:
  • PERT: Pre-training BERT with Permuted Language Model
  • github repo

哈工大讯飞联合实验室: Joint Laboratory of HIT and iFLYTEK Research (HFL).

Huawei

• PanGu (盘古) family
  • PanGu-Alpha
    • PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
    • 启智 AI协作平台
  • PanGu-Bot
    • PanGu-Bot: Efficient Generative Dialogue Pre-training from Pre-trained Language Model
  • PanGu-Coder
    • PanGu-Coder: Program Synthesis with Function-Level Language Modeling
• NEZHA (哪吒)
  • NEZHA: Neural Contextualized Representation for Chinese Language Understanding

Huawei-Noah offical github repo for Pretrained-Language Model.

Inspur

• Yuan 1.0
  • Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning
  • github repo

IDEA

• 封神榜——中文语言预训练模型开源计划
  • Erlangshen (二郎神)
  • Wenzhong (闻仲)
  • Rangdeng (燃灯)
  • Yuyuan (余元)
  • Bigan (比干)
  • Zhouwenwang (周文王)
  • Taiyi (太乙)

IDEA refers to 粤港澳大湾区数字经济研究院.

JD (JINGDONG)

• 织女 Vega v1

Langboat

• Mengzi
  • Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese
  • github repo

Microsoft

• Megatron-Turning NLG family
• DeBERTa series
  • DeBERTa: Decoding-enhanced BERT with Disentangled Attention
  • DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
  • github repo
  • Hugging Face 🤗
• DeepNMT
  • Very Deep Transformers for Neural Machine Translation
  • github repo
• DeepNet
  • DeepNet: Scaling Transformers to 1,000 Layers
• GODEL
  • GODEL: Large-Scale Pre-Training for Goal-Directed Dialog
  • github repo
• METRO
  • METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals

Nvidia

• Megatron-LM
  • Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
  • github repo

OpenAI

• GPT-family
  • GPT, GPT-2, GPT-3
    • GPT: Improving Language Understanding by Generative Pre-Training
    • GPT-2: Language Models are Unsupervised Multitask Learners
    • GPT-3: Language Models are Few-Shot Learners
    • GPT-2 github repo
    • GPT-2 Hugging Face 🤗
  • WebGPT
    • WebGPT: Browser-assisted question-answering with human feedback
  • InstructGPT
    • Training language models to follow instructions with human feedback
• Codex
  • Evaluating Large Language Models Trained on Code

OPPO

• OBERT

Tencent

• Motian (摩天)
• ShenZhou (神舟)
• ShenNonG (神农)
• HunYuan (混元)

THU (Tsinghua) & BAAI (Beijing)

• THUDM
  • Wenhui (文汇)
    • github repo
  • GLM-130B
    • github repo
    • demo for GLM-130B

THUDM refers to "Data Mining Research Group at Tsinghua University".

• Tsinghua AI

  • ERNIE
    • ERNIE: Enhanced Language Representation with Informative Entities
  • CPM family
    • CPM-1
      • CPM: A Large-scale Generative Chinese Pre-trained Language Model
      • github repo
    • CPM-2
      • CPM-2: Large-scale cost-effective pre-trained language models
      • github repo
      • 启智 AI协作平台

• THU-CoAI

  • EVA family
    • EVA 1.0
      • EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training
      • github project
    • EVA 2.0
      • EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Training
      • github project

THU-CoAI refers to "Conversational AI groups from Tsinghua University".

• BAAI
  • Chinese-Transformer-XL
    • github repo
    • 启智 AI协作平台
  • GLM
    • GLM: General Language Model Pretraining with Autoregressive Blank Infilling
    • 启智 AI协作平台

More PLMs are coming ...

🔥Coming soon!🔥

---

Claims

Thanks for all these companies and orgnizations paying a lot of money and efforts to build and train these large models for the benefit of all human beings.

This repo is inspired by awesome pretrained Chinese NLP models.