Colloquiums and Conferences
AI for Good: Recent Advances in Automatic Cued Speech Video Recognition and Generation
Time: 2024-05-23 16:26:00
Title: AI for Good: Recent Advances in Automatic Cued Speech Video Recognition and Generation
Speaker: Assistant Prof. Li Liu (Invited by Prof. Yannan Chen)
Affiliation: The Hong Kong University of Science and Technology (Guangzhou)
Time: May 22, 09:30–10:30
Venue: Lecture Hall, Room 111, West Building, School of Mathematical Sciences
Speaker Biography:
Li Liu is an assistant professor and doctoral supervisor at The Hong Kong University of Science and Technology (Guangzhou). She received her Ph.D. in September 2018 from GIPSA-lab, jointly affiliated with Université Grenoble Alpes and the French National Centre for Scientific Research (CNRS). Her research focuses on multimodal audio-visual speech recognition and generation, and medical artificial intelligence. As first or corresponding author she has published more than 40 papers in top journals and conferences including TPAMI, TMM, TMI, NeurIPS, ICCV, ECCV, ACM MM and ICASSP. She served as Local Chair (China site) for ICASSP 2022 and was selected under the Shenzhen Peacock Plan for overseas high-level talents. As principal investigator she has led projects funded by the NSFC Young Scientists Fund, the Guangdong Regional Joint Fund, the Alibaba Innovative Research Programme, the Tencent Sustainable Social Value Venture Programme and the Tencent AI Lab Rhino-Bird Programme. She received the 2017 Sephora Berribi Award for women scientists in mathematics and computer science in France, and the Shenzhen Association for Science and Technology Outstanding Paper Award in Artificial Intelligence in 2022 and 2023.
Abstract:
Automatic cued speech recognition and generation are primarily used for intelligent, accessible human–computer interaction. Cued speech uses lip shapes and hand gestures to encode spoken language and thereby support communication for people with hearing impairments; we were the first to propose a Chinese cued speech system. For cued speech recognition, we have recently proposed a low-rank Transformer based cross-modal mutual learning framework to facilitate multimodal interaction. Compared with traditional approaches, our model encodes modality-specific information through a modality-independent dictionary representation, yielding a shared multimodal representation of linguistic information. For cued speech generation, we propose a chain-of-thought prompted diffusion framework for cued speech video generation, using large language models and prompt engineering to capture the complex relationship between textual descriptions and gestural features, and improving the accuracy and diversity of generated gesture videos. In addition, we have built the first large-scale multi-speaker Chinese cued speech video dataset. Our methods achieve state-of-the-art recognition and generation performance on cued speech datasets in different languages (Chinese, French and English).