jaedukseo/encode-in-style

The official repository of "Encode-in-Style: Latent-based Video Encoding using StyleGAN2"

Encode-in-Style: Latent-based Video Encoding using StyleGAN2

Trevine Oorloff, Yaser Yacoob

We propose an end-to-end facial video encoding approach that facilitates data-efficient high-quality video re-synthesis by optimizing low-dimensional edits of a single Identity-latent. The approach builds on StyleGAN2 image inversion and multi-stage non-linear latent-space editing to generate videos that are nearly comparable to input videos. It economically captures face identity, head-pose, and complex facial motions at fine levels, and thereby bypasses training and person modeling which tend to hamper many re-synthesis approaches. The approach is designed with maximum data efficiency, where a single W+ latent and 35 parameters per frame enable high-fidelity video rendering. This pipeline can also be used for puppeteering (i.e., motion transfer).

Description

Official repository of the "Encode-in-Style: Latent-based Video Encoding using StyleGAN2" paper.
Note: The code would be released in the future.

Table of Contents

• Getting Started
  • Prerequisites
  • Installation
• Dataset
• Data Pre-processing
• Head-Pose Encoding
• Facial Attribute Encoding
• Rendering
• Generator Fine-tuning
• Evaluation
• Acknowledgments
• Bibtex