Training Quantized Deep Neural Networks via Cooperative Coevolution

Peng，Fu; Liu，Shengcai; Lu，Ning; Tang，Ke

10.1007/978-3-031-09726-3_8

2022

13th International Conference on Swarm Intelligence (ICSI)

0302-9743

1611-3349

978-3-031-09726-3

LECT NOTES ARTIF INT

13345 LNCS

81-93

JUL 15-19, 2022

null,Xian,PEOPLES R CHINA

GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND

SPRINGER INTERNATIONAL PUBLISHING AG

This work considers a challenging Deep Neural Network (DNN) quantization task that seeks to train quantized DNNs without involving any full-precision operations. Most previous quantization approaches are not applicable to this task since they rely on full-precision gradients to update network weights. To fill this gap, in this work we advocate using Evolutionary Algorithms (EAs) to search for the optimal low-bits weights of DNNs. To efficiently solve the induced large-scale discrete problem, we propose a novel EA based on cooperative coevolution that repeatedly groups the network weights based on the confidence in their values and focuses on optimizing the ones with the least confidence. To the best of our knowledge, this is the first work that applies EAs to train quantized DNNs. Experiments show that our approach surpasses previous quantization approaches and can train a 4-bit ResNet-20 on the Cifar-10 dataset with the same test accuracy as its full-precision counterpart.

Cooperative coevolution Evolutionary algorithm Large-scale discrete optimization Neural network quantization

First ; Corresponding

English

EI ; CPCI-S

Shenzhen Peacock Plan[KQTD2016112514355531] ; Guangdong Provincial Key Laboratory[2020B121201001]

Computer Science ; Robotics

Computer Science, Artificial Intelligence ; Robotics

WOS:000874477100008

20223012407916

Deep neural networks ; Optimization ; Statistical tests

Ergonomics and Human Factors Engineering:461.4 ; Optimization Techniques:921.5 ; Mathematical Statistics:922.2

2-s2.0-85134676021

Scopus

Guangdong Key Laboratory of Brain-Inspired Intelligent Computation,Department of Computer Science and Engineering,Southern University of Science and Technology,Shenzhen,518055,China

Peng，Fu,Liu，Shengcai,Lu，Ning,et al. Training Quantized Deep Neural Networks via Cooperative Coevolution[C]. GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND:SPRINGER INTERNATIONAL PUBLISHING AG,2022:81-93.