Model Compression by Iterative Pruning with Knowledge Distillation and Its Application to Speech Enhancement

Wei，Zeyuan1; Li，Hao2; Zhang，Xueliang1

10.21437/Interspeech.2022-619

2022

Interspeech Conference

2308-457X

1990-9772

Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH

2022-September

941-945

SEP 18-22, 2022

null,Incheon,SOUTH KOREA

C/O EMMANUELLE FOXONET, 4 RUE DES FAUVETTES, LIEU DIT LOUS TOURILS, BAIXAS, F-66390, FRANCE

ISCA-INT SPEECH COMMUNICATION ASSOC

Over the past decade, deep learning has demonstrated its effectiveness and keeps setting new records in a wide variety of tasks. However, good model performance usually leads to a huge amount of parameters and extremely high computational complexity which greatly limit the use cases of deep learning models, particularly in embedded systems. Therefore, model compression is getting more and more attention. In this paper, we propose a compression strategy based on iterative pruning and knowledge distillation. Specifically, in each iteration, we first utilize a pruning criterion to drop the weights which have less impact on performance. Then, the model before pruning is used as a teacher to fine-tune the student which is the model after pruning. After several iterations, we get the final compressed model. The proposed method is verified on gated convolutional recurrent network (GCRN) and long short-term memory (LSTM) for single channel speech enhancement tasks. Experimental results show that the proposed compression strategy can dramatically reduce the model size by 40x without significant performance degradation for GCRN.

knowledge distillation pruning quantization speech enhancement

Others

English

CPCI-S

Acoustics ; Audiology & Speech-Language Pathology ; Computer Science ; Engineering

Acoustics ; Audiology & Speech-Language Pathology ; Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic

WOS:000900724501024

2-s2.0-85140075848

Scopus

1.Department of Computer Science,Inner Mongolia University,Canada
2.Department of Electrical and Electronic Engineering,Southern University of Science and Technology,China

Wei，Zeyuan,Li，Hao,Zhang，Xueliang. Model Compression by Iterative Pruning with Knowledge Distillation and Its Application to Speech Enhancement[C]. C/O EMMANUELLE FOXONET, 4 RUE DES FAUVETTES, LIEU DIT LOUS TOURILS, BAIXAS, F-66390, FRANCE:ISCA-INT SPEECH COMMUNICATION ASSOC,2022:941-945.