Efficient Large Language Models: A Survey

Efficient Large Language Models: A Survey [arXiv]

Zhongwei Wan¹, Xin Wang¹, Che Liu², Samiul Alam¹, Yu Zheng³, Zhongnan Qu⁴, Shen Yan⁵, Yi Zhu⁶, Quanlu Zhang⁷, Mosharaf Chowdhury⁸, Mi Zhang¹

¹The Ohio State University, ²Imperial College London, ³Michigan State University, ⁴Amazon AWS AI, ⁵Google Research, ⁶Boson AI, ⁷Microsoft Research Asia, ⁸University of Michigan

📌 What is This Survey About?

Large Language Models (LLMs) have demonstrated remarkable capabilities in many important tasks and have the potential to make a substantial impact on our society. Such capabilities, however, come with considerable resource demands, highlighting the strong need to develop effective techniques for addressing the efficiency challenges posed by LLMs. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We hope our survey and this GitHub repository can serve as valuable resources to help researchers and practitioners gain a systematic understanding of the research developments in efficient LLMs and inspire them to contribute to this important and exciting field. We will actively maintain this repository and update the survey by incorporating new research as it emerges.

🤔 Why Efficient LLMs are Needed?

Although LLMs are leading the next wave of AI revolution, the remarkable capabilities of LLMs come at the cost of their substantial resource demands. Figure 1 (left) illustrates the relationship between model performance and the the GPU time required during training for LLaMA series. As shown, the amount of GPU time grows exponentially as the number of model parameter scales up. In addition to training, inference also contributes quite significantly to the operational cost of LLMs. As depicted in Figure 2 (right), more advanced LLMs exhibit higher memory usage and energy consumption during inference, presenting challenges for these models in expanding their reach to a broader customer base and diverse applications in a cost-effective way. With the rapid expansion of applications and the customer base for LLMs, the operational cost during inference in terms of energy consumption and memory usage would increase and exceed the training cost and become the dominant factor for the overall environmental impact.

📖 Table of Content

🤖 Model-Centric Methods
🔢 Data-Centric Methods
- Data Selection
  - Data Selection for Efficient Pre-Training
  - Data Selection for Efficient Fine-Tuning
- Prompt Engineering
🧑‍💻 System-Level Efficiency Optimization and LLM Frameworks
- System-Level Efficiency Optimization
- LLM Frameworks

🤖 Model-Centric Methods

Model Compression

Quantization

Post-Training Quantization

Weight-Only Quantization

GPTQ: Accurate Quantization for Generative Pre-trained Transformers, ICLR, 2023 [Paper] [Code]
QuIP: 2-Bit Quantization of Large Language Models With Guarantees, arXiv, 2023 [Paper] [Code]
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, arXiv, 2023 [Paper] [Code]
OWQ: Lessons Learned from Activation Outliers for Weight Quantization in Large Language Models, arXiv, 2023 [Paper] [Code]
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression, arXiv, 2023 [Paper] [Code]
FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs, NeurIPS-ENLSP, 2023 [Paper]
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, NeurlPS, 2022 [Paper] [Code]
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning, NeurIPS, 2022 [Paper] [Code]

Weight-Activation Co-Quantization

Intriguing Properties of Quantization at Scale, NeurIPS, 2023 [Paper]
ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation, arXiv, 2023 [Paper] [Code]
ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats, NeurIPS-ENLSP, 2023 [Paper] [Code]
OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization, ISCA, 2023 [Paper] [Code]
RPTQ: Reorder-based Post-training Quantization for Large Language Models, arXiv, 2023 [Paper] [Code]
Outlier Suppression+: Accurate Quantization of Large Language Models by Equivalent and Optimal Shifting and Scaling, arXiv, 2023 [Paper] [Code]
QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models, arXiv, 2023 [Paper]
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, ICML, 2023 [Paper] [Code]
ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers, NeurIPS, 2022 [Paper]

Quantization-Aware Training

BitNet: Scaling 1-bit Transformers for Large Language Models, arXiv, 2023 [Paper]
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models, arXiv, 2023 [Paper] [Code]
Compression of Generative Pre-trained Language Models via Quantization, ACL, 2022 [Paper]

Parameter Pruning

Structured Pruning

LLM-Pruner: On the Structural Pruning of Large Language Models, NeurIPS, 2023 [Paper] [Code]
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning, NeurIPS-ENLSP, 2023 [Paper] [Code]
LoRAPrune: Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning, arXiv, 2023 [Paper]
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks, NeurIPS, 2021 [Paper] [Code]

Unstructured Pruning

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot, ICML, 2023 [Paper] [Code]
A Simple and Effective Pruning Approach for Large Language Models, arXiv, 2023 [Paper] [Code]
One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models, arXiv, 2023 [Paper]

Low-Rank Approximation

TensorGPT: Efficient Compression of the Embedding Layer in LLMs based on the Tensor-Train Decomposition, arXiv, 2023 [Paper]
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation, ICML, 2023 [Paper] [Code]

Knowledge Distillation

White-Box KD

Towards the Law of Capacity Gap in Distilling Language Models, arXiv, 2023 [Paper] [Code]
Baby Llama: Knowledge Distillation from an Ensemble of Teachers Trained on a Small Dataset with no Performance Penalty, arXiv, 2023 [Paper]
Knowledge Distillation of Large Language Models, arXiv, 2023 [Paper] [Code]
GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models, arXiv, 2023 [Paper]
Propagating Knowledge Updates to LMs Through Distillation, arXiv, 2023 [Paper] [Code]
Less is More: Task-aware Layer-wise Distillation for Language Model Compression, ICML, 2023 [Paper]
Token-Scaled Logit Distillation for Ternary Weight Generative Language Models, arXiv, 2023 [Paper]

Black-Box KD

Instruction Tuning with GPT-4, arXiv, 2023 [Paper] [Code]
Lion: Adversarial Distillation of Closed-Source Large Language Model, arXiv, 2023 [Paper] [Code]
Specializing Smaller Language Models towards Multi-Step Reasoning, ICML, 2023 [Paper] [Code]
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes, ACL, 2023 [Paper]
Large Language Models Are Reasoning Teachers, ACL, 2023 [Paper] [Code]
SCOTT: Self-Consistent Chain-of-Thought Distillation, ACL, 2023 [Paper] [Code]
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step, ACL, 2023 [Paper]
Distilling Reasoning Capabilities into Smaller Language Models, ACL, 2023 [Paper] [Code]
In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models, arXiv, 2022 [Paper]
Explanations from Large Language Models Make Small Reasoners Better, arXiv, 2022 [Paper]
DISCO: Distilling Counterfactuals with Large Language Models, arXiv, 2022 [Paper] [Code]

Efficient Pre-Training

Mixed Precision Acceleration

GACT: Activation Compressed Training for Generic Network Architectures, ICML, 2022 [Paper] [Code]
Mesa: A Memory-saving Training Framework for Transformers, arXiv, 2021 [Paper] [Code]
Mixed Precision Training, ICLR, 2018 [Paper]

Scaling Models

Learning to Grow Pretrained Models for Efficient Transformer Training, ICLR, 2023 [Paper] [Code]
2x Faster Language Model Pre-training via Masked Structural Growth, arXiv, 2023 [Paper]
Reusing Pretrained Models by Multi-linear Operators for Efficient Training, NeurIPS, 2023 [Paper]
FLM-101B: An Open LLM and How to Train It with $100 K Budget, arXiv, 2023 [Paper] [Code]
Knowledge Inheritance for Pre-trained Language Models, NAACL, 2022 [Paper] [Code]
Staged Training for Transformer Language Models, ICML, 2022 [Paper] [Code]

Initialization Techniques

Deepnet: Scaling transformers to 1,000 layers, arXiv, 2022 [Paper] [Code]
ZerO Initialization: Initializing Neural Networks with only Zeros and Ones, TMLR, 2022 [Paper] [Code]
Rezero is All You Need: Fast Convergence at Large Depth, UAI, 2021 [Paper] [Code]
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks, NeurIPS, 2020 [Paper]
Improving Transformer Optimization Through Better Initialization, ICML, 2020 [Paper] [Code]
Fixup Initialization: Residual Learning without Normalization, ICLR, 2019 [Paper]
On Weight Initialization in Deep Neural Networks, arXiv, 2017 [Paper]

Optimization Strategies

Symbolic Discovery of Optimization Algorithms, arXiv, 2023 [Paper]
Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training, arXiv, 2023 [Paper] [Code]

Efficient Fine-Tuning

Parameter-Efficient Fine-Tuning

Adapter-based Tuning

OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models, ACL Demo, 2023 [Paper] [Code]
LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models, EMNLP, 2023 [Paper] [Code]
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers, NeurIPS, 2023 [Paper] [Code]
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning, NeurIPS, 2022 [Paper] [Code]
Meta-Adapters: Parameter Efficient Few-shot Fine-tuning through Meta-Learning, AutoML, 2022 [Paper]
AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning, EMNLP, 2022 [Paper] [Code]

Low-Rank Adaptation

LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning, arXiv, 2023 [Paper]
LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition, arXiv, 2023 [Paper] [Code]
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models, arXiv, 2023 [Paper] [Code]
Multi-Head Adapter Routing for Cross-Task Generalization, NeurIPS, 2023 [Paper] [Code]
Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning, ICLR, 2023 [Paper]
DyLoRA: Parameter-Efficient Tuning of Pretrained Models using Dynamic Search-Free Low Rank Adaptation, EACL, 2023 [Paper] [Code]
Tied-Lora: Enhacing Parameter Efficiency of LoRA with Weight Tying, arXiv, 2023 [Paper]
LoRA: Low-Rank Adaptation of Large Language Models, ICLR, 2022 [Paper] [Code]

Prefix Tuning

LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention, arXiv, 2023 [Paper] [Code]
Prefix-Tuning: Optimizing Continuous Prompts for Generation ACL, 2021 [Paper] [Code]

Prompt Tuning

GPT Understands, Too, AI Open, 2023 [Paper] [Code]
Multi-Task Pre-Training of Modular Prompt for Few-Shot Learning ACL, 2023 [Paper] [Code]
Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning, ICLR, 2023 [Paper]
PPT: Pre-trained Prompt Tuning for Few-shot Learning, ACL, 2022 [Paper] [Code]
Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers, EMNLP-Findings, 2022 [Paper] [Code]
P-Tuning v2: Prompt Tuning Can Be Comparable to Finetuning Universally Across Scales and Tasks，ACL-Short, 2022 [Paper] [Code]
The Power of Scale for Parameter-Efficient Prompt Tuning, EMNLP, 2021 [Paper]

Memory-Efficient Fine-Tuning

Memory-Efficient Selective Fine-Tuning, ICML Workshop, 2023 [Paper]
Full Parameter Fine-tuning for Large Language Models with Limited Resources, arXiv, 2023 [Paper] [Code]
Fine-Tuning Language Models with Just Forward Passes, NeurIPS, 2023 [Paper] [Code]
Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization, NeurIPS, 2023 [Paper]
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models, arXiv, 2023 [Paper] [Code]
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models, arXiv, 2023 [Paper] [Code]
QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS, 2023 [Paper] [Code1] [Code2]

Efficient Inference

Speculative Decoding

Accelerating Transformer Inference for Translation via Parallel Decoding, ACL, 2023 [Paper] [Code]
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads, Blog, 2023 [Blog] [Code]
Fast Inference from Transformers via Speculative Decoding, ICML, 2023 [Paper]
Accelerating LLM Inference with Staged Speculative Decoding, ICML Workshop, 2023 [Paper]
Accelerating Large Language Model Decoding with Speculative Sampling, arXiv, 2023 [Paper]
Speculative Decoding with Big Little Decoder, NeurIPS, 2023 [Paper] [Code]
SpecInfer: Accelerating Generative LLM Serving with Speculative Inference and Token Tree Verification, arXiv, 2023 [Paper] [Code]
Inference with Reference: Lossless Acceleration of Large Language Models, arXiv, 2023 [Paper] [Code]

KV-Cache Optimization

SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference, arXiv, 2023 [Paper]
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models, NeurIPS, 2023 [Paper]
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time, arXiv, 2023 [Paper]
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers, arXiv, 2023 [Paper]

Efficient Architecture

Efficient Attention

Sharing-based Attention

GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, EMNLP, 2023 [Paper]
Fast Transformer Decoding: One Write-Head is All You Need, arXiv, 2019 [Paper]

Feature Information Reduction

Nyströmformer: A nyström-based algorithm for approximating self-attention, AAAI, 2021 [Paper] [Code]
Funnel-transformer: Filtering out sequential redundancy for efficient language processing, NeurIPS, 2020 [Paper] [Code]
Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks, ICML, 2019 [Paper]

Kernelization or Low-Rank

Sumformer: Universal Approximation for Efficient Transformers, ICML Workshop, 2023 [Paper]
FLuRKA: Fast fused Low-Rank & Kernel Attention, arXiv, 2023 [Paper]
Scatterbrain: Unifying Sparse and Low-rank Attention, NeurlPS, 2021 [Paper] [Code]
Rethinking Attention with Performers, ICLR, 2021 [Paper] [Code]
Random Feature Attention, ICLR, 2021 [Paper]
Linformer: Self-Attention with Linear Complexity, arXiv, 2020 [Paper] [Code]
Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer, ICASSP, 2020 [Paper]
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention, ICML, 2020 [Paper] [Code]

Fixed Pattern Strategies

Faster Causal Attention Over Large Sequences Through Sparse Flash Attention, ICML Workshop, 2023 [Paper]
Poolingformer: Long Document Modeling with Pooling Attention, ICML, 2021 [Paper]
Big Bird: Transformers for Longer Sequences, NeurIPS, 2020 [Paper] [Code]
Longformer: The Long-Document Transformer, arXiv, 2020 [Paper] [Code]
Blockwise Self-Attention for Long Document Understanding, EMNLP, 2020 [Paper] [Code]

Learnable Pattern Strategies

HyperAttention: Long-context Attention in Near-Linear Time, arXiv, 2023 [Paper] [Code]
ClusterFormer: Neural Clustering Attention for Efficient and Effective Transformer, ACL, 2022 [Paper]
Reformer: The Efficient Transformer, ICLR, 2022 [Paper] [Code]
Sparse Sinkhorn Attention, ICML, 2020 [Paper]
Fast Transformers with Clustered Attention, NeurIPS, 2020 [Paper] [Code]
Efficient Content-Based Sparse Attention with Routing Transformers, TACL, 2020 [Paper] [Code]

Mixture of Experts

MoE-based LLMs

Mistral 7B, Mistral AI Blog, 2023 [Blog] [Code]
PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing, arXiv, 2023 [Paper]
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, JMLR, 2022 [Paper] [Code]
Efficient Large Scale Language Modeling with Mixtures of Experts, EMNLP, 2022 [Paper] [Code]
BASE Layers: Simplifying Training of Large, Sparse Models, ICML, 2021 [Paper] [Code]
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, ICLR, 2021 [Paper]

Algorithm-Level MoE Optimization

Lifelong Language Pretraining with Distribution-Specialized Experts, ICML, 2023 [Paper]
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models, arXiv, 2023 [Paper]
Mixture-of-Experts with Expert Choice Routing, NeurIPS, 2022 [Paper]
StableMoE: Stable Routing Strategy for Mixture of Experts, ACL, 2022 [Paper] [Code]
On the Representation Collapse of Sparse Mixture of Experts, NeurIPS, 2022 [Paper]
TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training, NeurIPS, 2022 [Paper] [Code]

Long Context LLMs

Extrapolation and Interpolation

A Length-Extrapolatable Transformer, ACL, 2023 [Paper] [Code]
Extending Context Window of Large Language Models via Positional Interpolation, arXiv, 2023 [Paper]
NTK Interpolation, Blog, 2023 [Reddit post]
YaRN: Efficient Context Window Extension of Large Language Models, arXiv, 2023 [Paper] [Code]
CLEX: Continuous Length Extrapolation for Large Language Models, arXiv, 2023 [Paper][Code]
Functional Interpolation for Relative Positions Improves Long Context Transformers, arXiv, 2023 [Paper]
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation, ICLR, 2022 [Paper] [Code]
Exploring Length Generalization in Large Language Models, NeurIPS, 2022 [Paper]
The EOS Decision and Length Extrapolation, EMNLP, 2020 [Paper] [Code]

Recurrent Structure

Recurrent Memory Transformer, NeurIPS, 2022 [Paper] [Code]
Block-Recurrent Transformers, NeurIPS, 2022 [Paper] [Code]
Memformer: A Memory-Augmented Transformer for Sequence Modeling, AACL-Findings, 2020 [Paper] [Code]
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context, ACL, 2019 [Paper] [Code]

Window & Stream Structure

Efficient Streaming Language Models with Attention Sinks, arXiv, 2023 [Paper] [Code]
Parallel Context Windows for Large Language Models, ACL, 2023 [Paper] [Code]
Naive Bayes-based Context Extension, GitHub, 2023 [Code]
LongNet: Scaling Transformers to 1,000,000,000 Tokens, arXiv, 2023 [Paper] [Code]
Efficient Long-Text Understanding with Short-Text Models, TACL, 2023 [Paper] [Code]
Structured Prompting: Scaling In-Context Learning to 1,000 Examples, arXiv, 2022 [Paper] [Code]

Memory-Retrieval Augmentation

Landmark Attention: Random-Access Infinite Context Length for Transformers, arXiv, 2023 [Paper] [Code]
Augmenting Language Models with Long-Term Memory, NeurIPS, 2023 [Paper]
Unlimiformer: Long-Range Transformers with Unlimited Length Input, NeurIPS, 2023 [Paper] [Code]
Focused Transformer: Contrastive Training for Context Scaling, NeurIPS, 2023 [Paper] [Code]
Retrieval meets Long Context Large Language Models, arXiv, 2023 [Paper]
Memorizing Transformers, ICLR, 2022 [Paper] [Code]

Transformer Alternative Architecture

State Space Models

Sparse Modular Activation for Efficient Sequence Modeling, NeurIPS, 2023 [Paper] [Code]
Mamba: Linear-Time Sequence Modeling with Selective State Spaces, arXiv, 2023 [Paper] [Code]
Hungry Hungry Hippos: Towards Language Modeling with State Space Models, ICLR 2023 [Paper] [Code]
Long Range Language Modeling via Gated State Spaces, ICLR, 2023 [Paper]
Block-State Transformers, NeurIPS, 2023 [Paper]
Efficiently Modeling Long Sequences with Structured State Spaces, ICLR, 2022 [Paper] [Code]
Diagonal State Spaces are as Effective as Structured State Spaces, NeurIPS, 2022 [Paper] [Code]

Other Sequential Models

RWKV: Reinventing RNNs for the Transformer Era, EMNLP-Findings, 2023 [Paper]
Hyena Hierarchy: Towards Larger Convolutional Language Models, arXiv, 2023 [Paper]
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers, arXiv, 2023 [Paper]

🔢 Data-Centric Methods

Data Selection

Data Selection for Efficient Pre-Training

Data Selection for Language Models via Importance Resampling, NeurIPS, 2023 [Paper] [Code]
NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework, ICML, 2022 [Paper] [Code]
Span Selection Pre-training for Question Answering, ACL, 2020 [Paper] [Code]

Data Selection for Efficient Fine-Tuning

Instruction Mining: When Data Mining Meets Large Language Model Finetuning, arXiv, 2023 [Paper]
Data-Efficient Finetuning Using Cross-Task Nearest Neighbors, ACL, 2023 [Paper] [Code]
Data Selection for Fine-tuning Large Language Models Using Transferred Shapley Values, ACL SRW, 2023 [Paper] [Code]
Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning, arXiv, 2023 [Paper]
AlpaGasus: Training A Better Alpaca with Fewer Data, arXiv, 2023 [Paper] [Code]
LIMA: Less Is More for Alignment, arXiv, 2023 [Paper]

Prompt Engineering

Few-Shot Prompting

Demonstration Organization

Demonstration Selection

Unified Demonstration Retriever for In-Context Learning, ACL, 2023 [Paper] [Code]
Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning, NeurIPS, 2023 [Paper] [Code]
In-Context Learning with Iterative Demonstration Selection, arXiv, 2022 [Paper]
Dr.ICL: Demonstration-Retrieved In-context Learning, arXiv, 2022 [Paper]
Learning to Retrieve In-Context Examples for Large Language Models, arXiv, 2022 [Paper]
Finding Supporting Examples for In-Context Learning, arXiv, 2022 [Paper]
Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering, ACL, 2023 [Paper] [Code]
Selective Annotation Makes Language Models Better Few-Shot Learners, ICLR, 2023 [Paper] [Code]
What Makes Good In-Context Examples for GPT-3? DeeLIO, 2022 [Paper]
Learning To Retrieve Prompts for In-Context Learning, NAACL-HLT, 2022 [Paper] [Code]
Active Example Selection for In-Context Learning, EMNLP, 2022 [Paper] [Code]
Rethinking the Role of Demonstrations: What makes In-context Learning Work? EMNLP, 2022 [Paper] [Code]

Demonstration Ordering

Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity, ACL, 2022 [Paper]
What Makes Good In-Context Examples for GPT-3? DeeLIO, 2022 [Paper]

Template Formatting

Instruction Generation

Large Language Models as Optimizers, arXiv, 2023 [Paper]
Instruction Induction: From Few Examples to Natural Language Task Descriptions, ACL, 2023 [Paper] [Code]
Large Language Models Are Human-Level Prompt Engineers, ICLR, 2023 [Paper] [Code]
TeGit: Generating High-Quality Instruction-Tuning Data with Text-Grounded Task Design, arXiv, 2023 [Paper]
Self-Instruct: Aligning Language Model with Self Generated Instructions, ACL, 2023 [Paper] [Code]

Multi-Step Reasoning

Automatic Chain of Thought Prompting in Large Language Models, ICLR, 2023 [Paper] [Code]
Measuring and Narrowing the Compositionality Gap in Language Models, EMNLP, 2023 [Paper] [Code]
ReAct: Synergizing Reasoning and Acting in Language Models, ICLR, 2023 [Paper] [Code]
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models, ICLR, 2023 [Paper]
Graph of Thoughts: Solving Elaborate Problems with Large Language Models, arXiv, 2023 [Paper] [Code]
Tree of Thoughts: Deliberate Problem Solving with Large Language Models, NeurIPS, 2023 [Paper] [Code]
Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR, 2023 [Paper]
Graph of Thoughts: Solving Elaborate Problems with Large Language Models, arXiv, 2023 [Paper] [Code]
Contrastive Chain-of-Thought Prompting, arXiv, 2023 [Paper] [Code]
Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generation, arXiv, 2023 [Paper]
Emergent Abilities of Large Language Models, TMLR, 2022 [Paper]

Prompt Compression

Learning to Compress Prompts with Gist Tokens, arXiv, 2023 [Paper]
Adapting Language Models to Compress Contexts, EMNLP, 2023 [Paper] [Code]
In-context Autoencoder for Context Compression in a Large Language Model, arXiv, 2023 [Paper] [Code]
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression, arXiv, 2023 [Paper] [Code]
Discrete Prompt Compression with Reinforcement Learning, arXiv, 2023 [Paper]
Nugget 2D: Dynamic Contextual Compression for Scaling Decoder-only Language Models, arXiv, 2023 [Paper]

Prompt Generation

Large Language Models Are Human-Level Prompt Engineers, ICLR, 2023 [Paper] [Code]
TempLM: Distilling Language Models into Template-Based Generators, arXiv, 2022 [Paper] [Code]
PromptGen: Automatically Generate Prompts using Generative Models, NAACL Findings, 2022 [Paper]
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts, EMNLP, 2020 [Paper] [Code]

🧑‍💻 System-Level Efficiency Optimization and LLM Frameworks

System-Level Efficiency Optimization

System-Level Pre-Training Efficiency Optimization

CoLLiE: Collaborative Training of Large Language Models in an Efficient Way, EMNLP, 2023 [Paper] [Code]
An Efficient 2D Method for Training Super-Large Deep Learning Models, IPDPS, 2023 [Paper] [Code]
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel, VLDB, 2023 [Paper]
Bamboo: Making Preemptible Instances Resilient for Affordable Training, NSDI, 2023 [Paper] [Code]
Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templates, SOSP, 2023 [Paper] [Code]
Varuna: Scalable, Low-cost Training of Massive Deep Learning Models, EuroSys, 2022 [Paper] [Code]
Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization, OSDI, 2022 [Paper] [Code]
Tesseract: Parallelize the Tensor Parallelism Efficiently, ICPP, 2022, [Paper]
Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning, OSDI, 2022, [Paper][Code]
Maximizing Parallelism in Distributed Training for Huge Neural Networks, arXiv, 2021 [Paper]
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, SC, 2021 [Paper] [Code]
ZeRO-Infinity: breaking the GPU memory wall for extreme scale deep learning, SC, 2021 [Paper]
ZeRO-Offload: Democratizing Billion-Scale Model Training, USENIX ATC, 2021 [Paper] [Code]
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, SC, 2020 [Paper] [Code]
PyTorch Distributed: Experiences on Accelerating Data Parallel Training, VLDB, 2020 [Paper]

System-Level Inference Efficiency Optimization

SAMP: A Model Inference Toolkit of Post-Training Quantization for Text Processing via Self-Adaptive Mixed-Precision, EMNLP, 2023 [Paper]
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU, ICML, 2023 [Paper] [Code]
SMDP-Based Dynamic Batching for Efficient Inference on GPU-Based Platforms, ICC, 2023 [Paper]
Flash-Decoding for Long-Context Inference, Blog, 2023 [Blog]
FlashDecoding++: Faster Large Language Model Inference on GPUs, arXiv, 2023 [Paper]
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time, ICML, 2023 [Paper]
Efficiently Scaling Transformer Inference, MLSys, 2023 [Paper]
S3: Increasing GPU Utilization during Generative Inference for Higher Throughput, arXiv, 2023 [Paper]
DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale, SC, 2022 [Paper]
Just-in-Time Dynamic-Batching, NeurIPS Workshop, 2018 [Paper]

System-Level Serving Efficiency Optimization

S-LoRA: Serving Thousands of Concurrent LoRA Adapters, arXiv, 2023 [Paper] [Code]
Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP, 2023 [Paper] [Code]
Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI, 2022 [Paper]
Fast Distributed Inference Serving for Large Language Models, arXiv, 2023 [Paper]
Chiplet Cloud: Building AI Supercomputers for Serving Large Generative Language Models, arXiv, 2023 [Paper]
SpotServe: Serving Generative Large Language Models on Preemptible Instances, arXiv, 2023 [Paper]
TurboTransformers: an efficient GPU serving system for transformer models, PPoPP, 2021 [Paper]

System-Level Efficient Architecture Optimization

System-Level Attention Optimization

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning, arXiv, 2023 [Paper] [Code]
Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP, 2023 [Paper] [Code]
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, NeurIPS, 2022 [Paper] [Code]
Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server, Nvidia Blog, 2022 [Blog]
A3: Accelerating Attention Mechanisms in Neural Networks with Approximation, HPCA, 2020 [Paper]

System-Level MoE Optimization

Tutel: Adaptive mixture-of-experts at scale, MLSys, 2023 [Paper] [Code]
MegaBlocks: Efficient Sparse Training with Mixture-of-Experts, MLSys, 2023 [Paper] [Code]
SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Offline and Online Parallelization, USENIX ATC, 2023 [Paper]
FastMoE: A Fast Mixture-of-Expert Training System, PPoPP, 2022 [Paper] [Code]
EdgeMoE: Fast On-Device Inference of MoE-based Large Language Models, arXiv, 2022 [Paper]
FasterMoE: modeling and optimizing training of large-scale dynamic pre-trained models, PPoPP, 2022 [Paper] [Code]
DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale, ICML, 2022 [Paper] [Code]

LLM Frameworks

	Efficient Training	Efficient Inference	Efficient Fine-Tuning
DeepSpeed [Code]	✅	✅	✅
Megatron [Code]	✅	✅	✅
Alpa [Code]	✅	✅	✅
ColossalAI [Code]	✅	✅	✅
FairScale [Code]	✅	✅	✅
Pax [Code]	✅	✅	✅
Composer [Code]	✅	✅	✅
vLLM [Code]	❌	✅	❌
Parallelformers [Code]	❌	✅	❌
OpenLLM [Code]	❌	✅	✅
Ray-LLM [Code]	❌	✅	❌
MLC-LLM [Code]	❌	✅	❌
Sax [Code]	❌	✅	❌
Mosec [Code]	❌	✅	❌
LLM-Foundry [Code]	✅	✅	❌

🖌️ Citation

If you find this survey useful to your work, please consider citing:

@misc{wan2023efficient,
      title={Efficient Large Language Models: A Survey}, 
      author={Zhongwei Wan and Xin Wang and Che Liu and Samiul Alam and Yu Zheng and Zhongnan Qu and Shen Yan and Yi Zhu and Quanlu Zhang and Mosharaf Chowdhury and Mi Zhang},
      year={2023},
      eprint={2312.03863},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

❤️ Contribution

This repository is maintained by tuidan ([email protected]), SUSTechBruce ([email protected]), samiul272 ([email protected]), and mi-zhang ([email protected]). We welcome feedback, suggestions, and contributions that can help improve this survey and repository so as to make them valuable resources to benefit the entire community.

If you have any suggestions regarding our taxonomy, find any missed papers, or update any preprint arXiv paper that has been accepted to some venue, feel free to send us an email or submit a pull request using the following markdown format.

Paper Title, <ins>Conference/Journal/Preprint, Year</ins>  [[pdf](link)] [[other resources](link)].

If one preprint paper has multiple versions, please use the earliest submitted year.
Display the papers in a year descending order (the latest, the first).

mard1no / efficient-llms-survey Goto Github PK

efficient-llms-survey's Introduction