LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT

Wang，Rui1; Bai，Qibing2; Ao，Junyi3; Zhou，Long4; Xiong，Zhixiang1; Wei，Zhihua1; Zhang，Yu2,5; Ko，Tom6; Li，Haizhou3

10.21437/Interspeech.2022-10269

2022

Interspeech Conference

2308-457X

1990-9772

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

2022-September

1686-1690

SEP 18-22, 2022

null,Incheon,SOUTH KOREA

C/O EMMANUELLE FOXONET, 4 RUE DES FAUVETTES, LIEU DIT LOUS TOURILS, BAIXAS, F-66390, FRANCE

ISCA-INT SPEECH COMMUNICATION ASSOC

Self-supervised speech representation learning has shown promising results in various speech processing tasks. However, the pre-trained models, e.g., HuBERT, are storage-intensive Transformers, limiting their scope of applications under low-resource settings. To this end, we propose LightHuBERT, a once-for-all Transformer compression framework, to find the desired architectures automatically by pruning structured parameters. More precisely, we create a Transformer-based supernet that is nested with thousands of weight-sharing subnets and design a two-stage distillation strategy to leverage the contextualized latent representations from HuBERT. Experiments on automatic speech recognition (ASR) and the SUPERB benchmark show the proposed LightHuBERT enables over 10 architectures concerning the embedding dimension, attention dimension, head number, feed-forward network ratio, and network depth. LightHuBERT outperforms the original HuBERT on ASR and five SUPERB tasks with the HuBERT size, achieves comparable performance to the teacher model in most tasks with a reduction of 29% parameters, and obtains a 3.5× compression ratio in three SUPERB tasks, e.g., automatic speaker verification, keyword spotting, and intent classification, with a slight accuracy loss. The code and pre-trained models are available at https://github.com/mechanicalsea/lighthubert.

knowledge distillation model compression neural architecture search speech pre-training Transformer

Others

English

CPCI-S

National Nature Science Foundation of China["61976160","61906137","61976158","62076184","62076182"] ; Shanghai Science and Technology Plan Project[21DZ1204800] ; Technology research plan project of Ministry of Public and Security[2020JSYJD01]

Acoustics ; Audiology & Speech-Language Pathology ; Computer Science ; Engineering

Acoustics ; Audiology & Speech-Language Pathology ; Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic

WOS:000900724501174

2-s2.0-85140048392

Scopus

1.Department of Computer Science and Technology,Tongji University,China
2.Department of Computer Science and Engineering,Southern University of Science and Technology,China
3.School of Data Science,The Chinese University of Hong Kong,Shenzhen,China
4.Microsoft,
5.Peng Cheng Laboratory,China
6.ByteDance AI Lab,

Wang，Rui,Bai，Qibing,Ao，Junyi,et al. LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT[C]. C/O EMMANUELLE FOXONET, 4 RUE DES FAUVETTES, LIEU DIT LOUS TOURILS, BAIXAS, F-66390, FRANCE:ISCA-INT SPEECH COMMUNICATION ASSOC,2022:1686-1690.