SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing

Ao, Junyi1,2; Wang, Rui3; Zhou, Long4; Wang, Chengyi4; Ren, Shuo4; Wu, Yu4; Liu, Shujie4; Ko, Tom1; Li, Qing2; Zhang, Yu1,5; Wei, Zhihua3; Qian, Yao4; Li, Jinyu4; Wei, Furu4

2022

60th Annual Meeting of the Association-for-Computational-Linguistics (ACL)

PROCEEDINGS OF THE 60TH ANNUAL MEETING OF THE ASSOCIATION FOR COMPUTATIONAL LINGUISTICS (ACL 2022), VOL 1: (LONG PAPERS)

MAY 22-27, 2022

null,Dublin,IRELAND

209 N EIGHTH STREET, STROUDSBURG, PA 18360 USA

ASSOC COMPUTATIONAL LINGUISTICS-ACL

Motivated by the success of T5 (Text-To-Text Transfer Transformer) in pre-trained natural language processing models, we propose a unified-modal SpeechT5 framework that explores the encoder-decoder pre-training for self-supervised speech/text representation learning. The SpeechT5 framework consists of a shared encoder-decoder network and six modal-specific (speech/text) pre/post-nets. After preprocessing the input speech/text through the pre-nets, the shared encoder-decoder network models the sequence-to-sequence transformation, and then the post-nets generate the output in the speech/text modality based on the output of the decoder. Leveraging large-scale unlabeled speech and text data, we pre-train SpeechT5 to learn a unified-modal representation, hoping to improve the modeling capability for both speech and text. To align the textual and speech information into this unified semantic space, we propose a cross-modal vector quantization approach that randomly mixes up speech/text states with latent units as the interface between encoder and decoder. Extensive evaluations show the superiority of the proposed SpeechT5 framework on a wide variety of spoken language processing tasks, including automatic speech recognition, speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification.

First

English

CPCI-S ; CPCI-SH

Computer Science ; Linguistics

Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Linguistics

WOS:000828702305058

Web of Science

1.Southern Univ Sci & Technol, Dept Comp Sci & Engn, Shenzhen, Peoples R China
2.Hong Kong Polytech Univ, Dept Comp, Hong Kong, Peoples R China
3.Tongji Univ, Dept Comp Sci & Technol, Shanghai, Peoples R China
4.Microsoft, Redmond, WA 98052 USA
5.Peng Cheng Lab, Shenzhen, Peoples R China

Ao, Junyi,Wang, Rui,Zhou, Long,et al. SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing[C]. 209 N EIGHTH STREET, STROUDSBURG, PA 18360 USA:ASSOC COMPUTATIONAL LINGUISTICS-ACL,2022.