Differentiable hierarchical and surrogate gradient search for spiking neural networks

Spiking neural network (SNN) has been viewed as a potential candidate for the nextgeneration of artificial intelligence with appealing characteristics such as sparsecomputation and inherent temporal dynamics. By adopting architectures of deepartificial neural networks (ANNs), SNNs are achieving competitive performancesin benchmark tasks such as image classification. However, successful architecturesof ANNs are not necessary ideal for SNN and when tasks become more diverseeffective architectural variations could be critical. To this end, we develop a spike-based differentiable hierarchical search (SpikeDHS) framework, where spike-basedcomputation is realized on both the cell and the layer level search space. Basedon this framework, we find effective SNN architectures under limited computationcost. During the training of SNN, a suboptimal surrogate gradient function couldlead to poor approximations of true gradients, making the network enter certainlocal minima. To address this problem, we extend the differential approach tosurrogate gradient search where the SG function is efficiently optimized locally.Our models achieve state-of-the-art performances on classification of CIFAR10/100and ImageNet with accuracy of 95.50%, 76.25% and 68.64%. On event-baseddeep stereo, our method finds optimal layer variation and surpasses the accuracyof specially designed ANNs meanwhile with 26× lower energy cost (6.7mJ),demonstrating the advantage of SNN in processing highly sparse and dynamicsignals.