vipl-slp/awesome-sign-language-processing

Resources about Sign Language Processing (e.g., Sign Language Recognition / Translation / Production)

Awesome Sign Language Processing

A curated list of sign language procesing (e.g., sign language recognition, sign language translation) and related area (e.g., speech translation, motion generation) resources. The old version can be found here.

• [Wiki] Sign Language
• [Wiki] Sign Language Recognition
• [Wiki] Gesture_recognition
• [Wiki] Sign Language Tranlsation
• [Resource] jinwchoi/awesome-action-recognition
• [Resource] xinghaochen/awesome-hand-pose-estimation
• [Resource] OpenHuman-ai/awesome-gesture_generation
• [Resource] zzw922cn/awesome-speech-recognition-speech-synthesis-papers

Contents

• Conference Paper
  • Sign Language Recognition
  • Sign Language Translation
  • Sign Language Production
  • Other Sign Language Topic
  • Co-speech Gesture Generation
  • Gesture Recognition
• Sign Language Workshop
• Dataset

Conference Paper

Sign Language Recognition

• [CVPR 2024] SignGraph: A Sign Sequence is Worth Graphs of Nodes. [code]
  Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Hongkai Wen, Kang Xia, Lei Xie, Sanglu Lu
• [EMNLP Findings 2023] Handshape-Aware Sign Language Recognition: Extended Datasets and Exploration of Handshape-Inclusive Methods. [paper]
  Xuan Zhang, Kevin Duh
• [EMNLP Findings 2023] Making Body Movement in Sign Language Corpus Accessible for Linguists and Machines with Three-Dimensional Normalization of MediaPipe. [paper]
  Houda Bouamor, Juan Pino, Kalika Bali
• [EMNLP Findings 2023] Linguistically Motivated Sign Language Segmentation. [paper]
  Amit Moryossef, Zifan Jiang, Mathias Müller, Sarah Ebling, Yoav Goldberg
• [NeurIPS 2023 Dataset] ASL Citizen: A Community-Sourced Dataset for Advancing Isolated Sign Language Recognition.[paper]
  Aashaka Desai, Lauren Berger, Fyodor O Minakov, Vanessa Milan, Chinmay Singh, Kriston L Pumphrey, Richard Ladner, Hal Daumé III, Alex Xijie Lu, Naomi Caselli, Danielle Bragg
• [NeurIPS 2023 Dataset] PopSign ASL v1.0: An Isolated American Sign Language Dataset Collected via Smartphones. [paper]
  Thad Starner, Sean Forbes, Matthew So, David Martin, Rohit Sridhar, Gururaj Deshpande, Sam Sepah, Sahir Shahryar, Khushi Bhardwaj, Tyler Kwok, Daksh Sehgal, Saad Hassan, Bill Neubauer, Sofia Anandi Vempala, Alec Tan, Jocelyn Heath, Unnathi Utpal Kumar, Priyanka Vijayaraghavan Mosur, Tavenner M. Hall, Rajandeep Singh et al.
• [ICCV 2023] CoSign: Exploring Co-occurrence Signals in Skeleton-based Continuous Sign Language Recognition.[paper]
  Peiqi Jiao, Yuecong Min, Yanan Li, Xiaotao Wang, Lei Lei, Xilin Chen
• [ICCV 2023] Human Part-wise 3D Motion Context Learning for Sign Language Recognition.[paper]
  Taeryung Lee, Yeonguk Oh and Kyoung Mu Lee
• [ICCV 2023] Improving Continuous Sign Language Recognition with Cross-Lingual Signs.[paper]
  Fangyun Wei, Yutong Chen
• [ICCV 2023] C2ST: Cross-modal Contextualized Sequence Transduction for Continuous Sign Language Recognition.[paper]
  Huaiwen Zhang, Zihang Guo, Yang Yang, Xin Liu, De Hu
• Complete List of Sign Language Recognition

Sign Language Translation

• [CVPR 2024] Representing Signs as Language: A New Method for Sign Language Translation from Videos.
  Jia Gong, Lin Geng Foo, Yixuan He, Hossein Rahmani , Jun Liu
• [ICLR 2024] Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation. [paper] Ryan Wong, Necati Cihan Camgoz, Richard Bowden
• [EMNLP Findings 2023] JWSign: A Highly Multilingual Corpus of Bible Translations for more Diversity in Sign Language Processing. [paper]
  Shester Gueuwou, Sophie Siake, Colin Leong, Mathias Müller
• [EMNLP Findings 2023] Cross-modality Data Augmentation for End-to-End Sign Language Translation. [paper]
  Jinhui Ye, Wenxiang Jiao, Xing Wang, Zhaopeng Tu, Hui Xiong
• [OpenReview] Towards Faithful Sign Language Translation. [paper]
  Yuecong Min, Xilin Chen
• [NeurIPS 2023 Dataset] YouTube-ASL: A Large-Scale, Open-Domain American Sign Language-English Parallel Corpus. [paper]
  David Uthus, Garrett Tanzer, Manfred Georg
• [NeurIPS 2023 Dataset] Auslan-Daily: Australian Sign Language Translation for Daily Communication and News. [paper]
  Xin Shen, Shaozu Yuan, Hongwei Sheng, Heming Du, Xin Yu
• [ICCV 2023] Sign Language Translation with Iterative Prototype.[paper]
  Huijie Yao, Wengang Zhou, Hao Feng, Hezhen Hu, Hao Zhou, Houqiang Li
• [ICCV 2023] Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining.[paper]
  Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, Du Zhang
• Complete List of Sign Language Translation

Sign Language Production

• [CVPR 2024] Neural Sign Actors: A diffusion model for 3D sign language production from text. [paper]
  Vasileios Baltatzis, Rolandos Alexandros Potamias, Evangelos Ververas, Guanxiong Sun, Jiankang Deng, Stefanos Zafeiriou
• [WACV 2024] Sign Language Production With Latent Motion Transformer. [paper]
  Pan Xie, Taiying Peng, Yao Du, Qipeng Zhang
• [CVPR 2022] Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production.[paper]
  Ben Saunders, Necati Cihan Camgoz, Richard Bowden
• [ICCV 2021] Mixed SIGNals: Sign Language Production via a Mixture of Motion Primitives.[paper]
  Ben Saunders; Necati Cihan Camgoz; Richard Bowden
• Complete List of Sign Language Production

Other Sign Language Topic

• [EMNLP Findings 2023] Enhancing Accessible Communication: from European Portuguese to Portuguese Sign Language. [paper]
  Catarina Sousa, Luisa Coheur, Mara Moita
• [CVPR 2023] CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning.[paper][code]
  Yiting Cheng, Fangyun Wei, Jianmin Bao, Dong Chen, Wenqiang Zhang
• [CVPR 2023] Ham2Pose: Animating Sign Language Notation into Pose Sequences. [paper]
  Rotem Shalev Arkushin, Amit Moryossef, Ohad Fried

Co-speech Gesture Generation

• [CVPR 2024] EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling.[paper]
  Haiyang Liu, Zihao Zhu, Giorgio Becherini, YICHEN PENG, Mingyang Su, YOU ZHOU, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, Michael J. Black
• [CVPR 2024] Weakly-Supervised Emotion Transition Learning for Diverse 3D Co-speech Gesture Generation.[paper]
  Xingqun Qi, Jiahao Pan, Peng Li, Ruibin Yuan, Xiaowei Chi, Mengfei Li, Wenhan Luo, Wei Xue, Shanghang Zhang, Qifeng Liu, Yike Guo
• [CVPR 2024] Towards Variable and Coordinated Holistic Co-Speech Motion Generation.
  Yifei Liu, Qiong Cao, Yandong Wen, Huaiguang Jiang, Changxing Ding
• [CVPR 2024] ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis.
  Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Marc Habermann, Christian Theobalt
• [CVPR 2024] Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model.
  Xu He, Qiaochu Huang, Zhensong Zhang, Zhiwei Lin, Zhiyong Wu, Sicheng Yang, Minglei Li, Zhiyi Chen, Songcen Xu, Xiaofei Wu
• [ICCV 2023] Continual Learning for Personalized Co-speech Gesture Generation.[paper]
  Chaitanya Ahuja, Pratik Joshi, Ryo Ishii, Louis-Philippe Morency
• [ICCV 2023] LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generation.[paper]
  Yihao Zhi, Xiaodong Cun, Xuelin Chen, Xi Shen, Wen Guo, Shaoli Huang, Shenghua Gao
• [CVPR 2023] QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generation.[paper]
  SiCheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Haolin Zhuang
• [CVPR 2023] Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generation.[paper]
  Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, Lequan Yu
• [CVPR 2023] Diverse 3D Hand Gesture Prediction from Body Dynamics by Bilateral Hand Disentanglement.[paper]
  Xingqun Qi, Chen Liu, Muyi Sun, Lincheng Li, Changjie Fan, Xin Yu
• [CVPR 2023] Co-speech Gesture Synthesis by Reinforcement Learning with Contrastive Pre-trained Rewards.[paper]
  Mingyang Sun, Mengchen Zhao, Yaqing Hou, Minglei Li, Huang Xu, Songcen Xu, Jianye HAO
• Complete List of Co-speech Gesture Generation

Gesture Recognition

• [ICCV 2023] Learning Robust Representations with Information Bottleneck and Memory Network for RGB-D-based Gesture Recognition.[paper]
  Yunan Li, Huizhou Chen, Guanwen Feng, Qiguang Miao
• [ICCV 2023] Data-Free Class-Incremental Hand Gesture Recognition.[paper]
  Shubhra Aich, Jesus Ruiz-Santaquiteria, Zhenyu Lu, Prachi Garg, K J Joseph, Alvaro Fernandez Garcia, Vineeth N Balasubramanian, Kenrick Kin, Chengde Wan, Necati Cihan Camgoz, Shugao Ma, Fernando De la Torre
• [BMVC 2022] Towards more efficient few-shot learning based human gesture recognition via dynamic vision sensors.[paper]
  Linglin Jing, Yifan Wang, Tailin Chen, Shirin Dora, Zhigang Ji, Hui Fang
• [BMVC 2022] Continuous Hand Gesture Recognition using Deep Coarse and Fine Hand Features.[paper]
  Hazem Wannous, Jean-Philippe Vandeborre
• [CVPR 2022] LD-ConGR: A Large RGB-D Video Dataset for Long-Distance Continuous Gesture Recognition.[paper]
  Dan Liu; Libo Zhang; Yanjun Wu
• [AAAI 2022] Learning Unseen Emotions from Gestures via Semantically-Conditioned Zero-Shot Perception with Adversarial Autoencoders.[paper]
  Abhishek Banerjee, Uttaran Bhattacharya, Aniket Bera
• Complete List of Gesture Recognition

Sign Language Workshop

• [ICCV ACVR 2023] Eleventh International Workshop on Assistive Computer Vision and Robotics
• [ECCV SLRTP 2020] Sign language recognition, translation & production. [Accepted papers]

Dataset

Dataset		Task		Modality		Statistic
Name	Language	Isolated	Continuous	RGB	Depth	#Signers	#Vocab	#Sequences
DGS Kinect	DGS	√		√	√	15	40	3,000
ASLLVD	ASL	√		√		6	2,742	9,794
DEVISIGN-L	CSL	√		√	√	8	2,000	24,000
CSL-500	CSL	√		√		50	500	125,000
MSASL	ASL	√		√		222	1,000	25,513
WLASL2000	ASL	√		√		119	2,000	21,083
AUTSL	TSL	√		√	√	43	226	38,336
SIGNUM	DGS		√	√	√	9	455	2,340
Phoenix14	DGS		√	√		9	1,200	6,841
KETI	KSL		√	√		14	524	15,672
GSL	GSL		√	√	√	7	310	10,290
CSL-100	CSL		√	√		50	2,000	25,000
CSL-Daily	CSL		√	√		10	2,000	20,654
BSL-1K	BSL		√	√		40	1,064	273K
VRT-NEWS	VGT		√	√		9	6,325	7,174
How2Sign	BSL		√	√	√	11	15,686	35,191
BOBSL	BSL		√	√		39	2,281	452K
OpenASL	ASL		√	√		220	33,549	98,417
YouTube-ASL	ASL		√	√		>2519	60K	11,093