cmmclee/awesome_talking_face_generation

Awesome talking face generation

papers & codes

2023

title	-	paper	code	dataset	keywords
DINet: Deformation Inpainting Network for Realistic Face Visually Dubbing on High Resolution Video	AAAI(23)	paper	code
CodeTalker: Speech-Driven 3D Facial Animation with Discrete Motion Prior		paper	code		3D
GENEFACE: GENERALIZED AND HIGH-FIDELITY AUDIO-DRIVEN 3D TALKING FACE SYNTHESIS	ICLR (23)	paper	code		NeRF
OPT: ONE-SHOT POSE-CONTROLLABLE TALKING HEAD GENERATION		paper
LipNeRF: What is the right feature space to lip-sync a NeRF?		paper			NeRF
Audio-Visual Face Reenactment	WACV (23)	paper	code
Towards Generating Ultra-High Resolution Talking-Face Videos With Lip Synchronization	WACV (23)	paper	-
StyleTalk: One-shot Talking Head Generation with Controllable Speaking Styles	AAAI(23)	paper	code
DiffTalk: Crafting Diffusion Models for Generalized Talking Head Synthesis		paper	proj		Diffusion
Diffused Heads: Diffusion Models Beat GANs on Talking-Face Generation		paper	proj		Diffusion
Speech Driven Video Editing via an Audio-Conditioned Diffusion Model		paper	code		Diffusion

2022

title	-	paper	code	dataset	keywords
Talking Head Generation with Probabilistic Audio-to-Visual Diffusion Priors		paper	proj
SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animation	cvpr(23)	paper	code
Compressing Video Calls using Synthetic Talking Heads	BMVC (22)	paper			application
EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion Model	SIGGRAPH (22)	paper			emotion
Learning Dynamic Facial Radiance Fields for Few-Shot Talking Head Synthesis	ECCV(22)	paper	code
Expressive Talking Head Generation with Granular Audio-Visual Control	CVPR(22)	paper	-
Talking Face Generation With Multilingual TTS	CVPR(22)	paper	code	-	-
Deep Learning for Visual Speech Analysis: A Survey	-	paper	-	-	survey
StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGAN	-	paper	code	-	stylegan
Semantic-Aware Implicit Neural Audio-Driven Video Portrait Generation	ECCV(22)	paper	code(coming soon)		NeRF
Cross-Modal Mutual Learning for Audio-Visual Speech Recognition and Manipulation	-	paper	-	-	-
SyncTalkFace: Talking Face Generation with Precise Lip-syncing via Audio-Lip Memory	AAAI(22)	paper(temp)	-	LRW, LRS2, BBC News	-
DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering		paper			NeRF
Face-Dubbing++: Lip-Synchronous, Voice Preserving Translation of Videos		paper
Dynamic Neural Textures: Generating Talking-Face Videos with Continuously Controllable Expressions		paper
DialogueNeRF: Towards Realistic Avatar Face-to-face Conversation Video Generation		paper
Talking Head Generation Driven by Speech-Related Facial Action Units and Audio- Based on Multimodal Representation Fusion		paper
StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generation	-	paper	-	-	-
AUTOLV: AUTOMATIC LECTURE VIDEO GENERATOR		paper
Synthesizing Photorealistic Virtual Humans Through Cross-modal Disentanglement		paper

2021

title	-	paper	code	dataset
Depth-Aware Generative Adversarial Network for Talking Head Video Generation		paper	code
		paper	code
Parallel and High-Fidelity Text-to-Lip Generation		paper
[Survey]Deep Person Generation: A Survey from the Perspective of Face, Pose and Cloth Synthesis	-	paper	-	-
FaceFormer: Speech-Driven 3D Facial Animation with Transformers	CVPR(22)	paper	code	-
Voice2Mesh: Cross-Modal 3D Face Model Generation from Voices	-	paper	code	-
FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning	ICCV	paper	code	-
Imitating Arbitrary Talking Style for Realistic Audio-Driven Talking Face Synthesis	-	paper	code	-
Audio-Driven Emotional Video Portraits	CVPR	paper	code	MEAD, LRW
LipSync3D: Data-Efficient Learning of Personalized 3D Talking Faces from Video using Pose and Lighting Normalization	CVPR	paper	-	-
Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation	CVPR	paper	code	VoxCeleb2, LRW
Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset	CVPR	paper	code	HDTF
MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement	ICCV	paper	code(coming soon)	-
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis	ICCV	paper	code	-
Write-a-speaker: Text-based Emotional and Rhythmic Talking-head Generation	AAAI	paper	code(coming soon)	Mocap dataset
Visual Speech Enhancement Without A Real Visual Stream	-	paper	-	-
Text2Video: Text-driven Talking-head Video Synthesis with Phonetic Dictionary	-	paper	code	-
Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motion	IJCAI	paper	code	VoxCeleb, GRID, LRW
3D-TalkEmo: Learning to Synthesize 3D Emotional Talking Head	-	paper	-	-
AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person	-	paper	-	VoxCeleb2, Obama

2020

title	-	paper	code	dataset
[Survey]What comprises a good talking-head video generation?: A survey and benchmark	-	paper	code	-
One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing	CVPR(21)	paper	code	-
Speech Driven Talking Face Generation from a Single Image and an Emotion Condition	-	paper	code	CREMA-D
A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild	ACMMM	paper	code	LRS2
Talking-head Generation with Rhythmic Head Motion	ECCV	paper	code	Crema, Grid, Voxceleb, Lrs3
MEAD: A Large-scale Audio-visual Dataset for Emotional Talking-face Generation	ECCV	paper	code	VoxCeleb2, AffectNet
Neural voice puppetry:Audio-driven facial reenactment	ECCV	paper	-	-
Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatars	ECCV	paper	code	-
HeadGAN:Video-and-Audio-Driven Talking Head Synthesis	-	paper	-	VoxCeleb2
MakeItTalk: Speaker-Aware Talking Head Animation	-	paper	code, code	VoxCeleb2, VCTK
Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose	-	paper	code	ImageNet, FaceWarehouse, LRW
Photorealistic Lip Sync with Adversarial Temporal Convolutional Networks	-	paper	-	-
SPEECH-DRIVEN FACIAL ANIMATION USING POLYNOMIAL FUSION OF FEATURES	-	paper	-	LRW
Animating Face using Disentangled Audio Representations	WACV	paper	-
Everybody’s Talkin’: Let Me Talk as You Want	-	paper	-	-
Multimodal Inputs Driven Talking Face Generation With Spatial-Temporal Dependency	-	paper	-	-
Speech Driven Talking Face Generation from a Single Image and an Emotion Condition	-	paper	-	-

2019

title	-	paper	code	dataset
Hierarchical Cross-Modal Talking Face Generation with Dynamic Pixel-Wise Loss	CVPR	paper	code	VGG Face, LRW

datasets

• MEAD link
• HDTF link
• CREMA-D link
• VoxCeleb link
• LRS2 link
• LRW link
• GRID link
• BIWI link
• SAVEE link

metrics

• PSNR (peak signal-to-noise ratio)
• SSIM (structural similarity index measure)
• LMD (landmark distance error)
• LRA (lip-reading accuracy) -
• FID (Fréchet inception distance)
• LSE-D (Lip Sync Error - Distance)
• LSE-C (Lip Sync Error - Confidence)
• LPIPS (Learned Perceptual Image Patch Similarity) -
• NIQE (Natural Image Quality Evaluator) -