An Integer-Only and Group-Vector Systolic Accelerator for Efficiently Mapping Vision Transformer on Edge

Huang, Mingqiang1; Luo, Junyi2; Ding, Chenchen2; Wei, Zikun2; Huang, Sixiao2; Yu, Hao2

2023-10-01

10.1109/TCSI.2023.3312775

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I-REGULAR PAPERS   Impact Factor & Quartile

1549-8328

1558-0806

Transformer-like network has shown remarkable high performance in both natural language processing and computer vision. However, the huge computational demands in non-linear floating-point arithmetic and the irregular memory access requirement in self-attention mechanism make it still a challenge to deploy Transformer on edge. To address the above issues, we propose integer-only quantization scheme for the simplification of non-linear operations (such as LayerNorm, Softmax and Gelu), meanwhile algorithm-hardware co-design strategy is applied to guarantee both the high accuracy and high efficiency. Besides, we construct general-purpose group vector systolic array to efficiently accelerate the matrix multiplication operations including both regular matrix-multiplication/convolution and the irregular multi-head self-attention mechanism. Unified data-package strategy and flexible on-/off-chip data storage management strategy are also proposed to further improve the performance. The design has been deployed on Xilinx ZCU102 FPGA platform, achieving an overall inference latency of 4.077ms and 11.15ms per image for ViT-tiny and ViT-s, respectively. The average throughput can reach as high as 762.7 GOPs, which shows significant improvement over the previous state-of-the-art FPGA Transformer accelerator.

Integer-only transformer systolic accelerator

SCI

English

Corresponding

STI 2030-Major Project[2022ZD0210600] ; National Natural Science Foundation of China (NSFC)[62034007] ; Shenzhen Science and Technology Program["KQTD20200820113051096","JCYJ20200109115210307"]

Engineering

Engineering, Electrical & Electronic

WOS:001092405100001

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC

ENGINEERING

Web of Science

1.Chinese Acad Sci, Shenzhen Inst Adv Technol, Shenzhen 518055, Peoples R China
2.Southern Univ Sci & Technol, Sch Microelect, Shenzhen 518055, Peoples R China

Huang, Mingqiang,Luo, Junyi,Ding, Chenchen,et al. An Integer-Only and Group-Vector Systolic Accelerator for Efficiently Mapping Vision Transformer on Edge[J]. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I-REGULAR PAPERS,2023.

Huang, Mingqiang,Luo, Junyi,Ding, Chenchen,Wei, Zikun,Huang, Sixiao,&Yu, Hao.(2023).An Integer-Only and Group-Vector Systolic Accelerator for Efficiently Mapping Vision Transformer on Edge.IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I-REGULAR PAPERS.

Huang, Mingqiang,et al."An Integer-Only and Group-Vector Systolic Accelerator for Efficiently Mapping Vision Transformer on Edge".IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS I-REGULAR PAPERS (2023).