BookKD: A novel knowledge distillation for reducing distillation costs by decoupling knowledge generation and learning

Zhu，Songling1; Shang，Ronghua1; Tang，Ke2; Xu，Songhua3; Li，Yangyang1

2023-11-04

10.1016/j.knosys.2023.110916

Knowledge-Based Systems   Impact Factor & Quartile

0950-7051

1872-7409

Knowledge distillation guides student networks’ training and enhances their performance through excellent teacher networks. However, along with the performance advantages, knowledge distillation also entails a huge computational burden, sometimes tens or even hundreds of times that of traditional training methods. So, this paper designs a book-based knowledge distillation (BookKD) to minimize the costs of knowledge distillation while improving performance. First, a decoupling-based knowledge distillation framework is designed. By decoupling the traditional knowledge distillation process into two independent sub-processes, book-making and book-learning, knowledge distillation can be completed with little resource consumption. Second, a book-making method based on knowledge ensemble and knowledge regularization is developed, which makes books by organizing and processing the knowledge generated by teachers. These books can replace these teachers to provide sufficient knowledge with little distillation costs. Finally, a book-learning method based on entropy dynamic adjustment and label smoothing is designed. The entropy dynamic adjustment optimizes the training loss and mitigates student networks’ difficulty in learning books. Label smoothing alleviates the student network's over-confidence in ground truth labels, which increases its attention to the class similarity knowledge in books. BookKD is tested on three image classification datasets, CIFAR100, ImageNet and ImageNet100, and an object detection dataset PASCAL VOC 2007. The experiment results indicate the advantages of BookKD in reducing distillation costs and improving distillation performance.

Image classification Knowledge distillation Knowledge ensemble Label smoothing Resource consumption

SCI

English

Others

National Natural Science Foundation of China[61871306];National Natural Science Foundation of China[62176200];

Computer Science

Computer Science, Artificial Intelligence

WOS:001080612800001

ELSEVIER

COMPUTER SCIENCE

2-s2.0-85170571218

Scopus

1.Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education,School of Artificial Intelligence,Xidian University,Xi'an,Shaanxi Province,710071,China
2.The Research Institute of Trustworthy Autonomous Systems,Southern University of Science and Technology,Shenzhen,518055,China
3.The Institute of Medical Artificial Intelligence,the Second Affiliated Hospital of Xi'an Jiaotong University,Xi'an,710004,China

Zhu，Songling,Shang，Ronghua,Tang，Ke,et al. BookKD: A novel knowledge distillation for reducing distillation costs by decoupling knowledge generation and learning[J]. Knowledge-Based Systems,2023,279.

Zhu，Songling,Shang，Ronghua,Tang，Ke,Xu，Songhua,&Li，Yangyang.(2023).BookKD: A novel knowledge distillation for reducing distillation costs by decoupling knowledge generation and learning.Knowledge-Based Systems,279.

Zhu，Songling,et al."BookKD: A novel knowledge distillation for reducing distillation costs by decoupling knowledge generation and learning".Knowledge-Based Systems 279(2023).