Discrete time convolution for fast event-based stereo

Kaixuan Zhang1,3; Kaiwei Che2,3; Jianguo Zhang1,4; Jie Cheng3; Ziyang Zhang3; Qinghai Guo3; Luziwei Leng3

10.1109/CVPR52688.2022.00848

2022

Conference on Computer Vision and Pattern Recognition (CVPR)

1063-6919

978-1-6654-6947-0

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

2022-June

8666-8676

18-24 June 2022

New Orleans, LA, USA

10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA

IEEE COMPUTER SOC

Inspired by biological retina, dynamical vision sensor transmits events of instantaneous changes of pixel intensity, giving it a series of advantages over traditional frame-based camera, such as high dynamical range, high temporal resolution and low power consumption. However, extracting information from highly asynchronous event data is a challenging task. Inspired by continuous dynamics of biological neuron models, we propose a novel encoding method for sparse events - continuous time convolution (CTC) - which learns to model the spatial feature of the data with intrinsic dynamics. Adopting channel-wise parameterization, temporal dynamics of the model is synchronized on the same feature map and diverges across different ones, enabling it to embed data in a variety of temporal scales. Abstracted from CTC, we further develop discrete time convolution (DTC) which accelerates the process with lower computational cost. We apply these methods to event-based multi-view stereo matching where they surpass state-of-the-art methods on benchmark criteria of the MVSEC dataset. Spatially sparse event data often leads to inaccurate estimation of edges and local contours. To address this problem, we propose a dual-path architecture in which the feature map is complemented by underlying edge information from original events extracted with spatially-adaptive denormalization. We demonstrate the superiority of our model in terms of speed (up to 110 FPS), accuracy and robustness, showing a great potential for real-time fast depth estimation. Finally, we perform experiments on the recent DSEC dataset to demonstrate the general usage of our model.

3D From Multi-view And Sensors Low-level Vision Vision + X

First

English

EI ; CPCI-S

National Key Research and Development Program of China[2021YFF1200800]

Computer Science ; Imaging Science & Photographic Technology

Computer Science, Artificial Intelligence ; Imaging Science & Photographic Technology

WOS:000870759101070

20224613120585

IEEE

1.Department of Computer Science and Engineering, Southern University of Science and Technology, China
2.Department of Electrical and Electronic Engineering, Southern University of Science and Technology, China
3.ACS Lab, Huawei Technologies, Shenzhen, China
4.Peng Cheng Lab, Shenzhen, China

Kaixuan Zhang,Kaiwei Che,Jianguo Zhang,et al. Discrete time convolution for fast event-based stereo[C]. 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA:IEEE COMPUTER SOC,2022:8666-8676.