Visual-tactile Sensing for Real-time Liquid Volume Estimation in Grasping

Fan Zhu1; Ruixing Jia1; Lei Yang1; Youcan Yan2; Zheng Wang3; Jia Pan1; Wenping Wang1

10.1109/IROS47612.2022.9981153

2022

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

2153-0858

978-1-6654-7928-8

2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

12542-12549

23-27 Oct. 2022

Kyoto, Japan

345 E 47TH ST, NEW YORK, NY 10017 USA

IEEE

We propose a deep visuo-tactile model for real-time estimation of the liquid inside a deformable container in a proprioceptive way. We fuse two sensory modalities, i.e., the raw visual inputs from the RGB camera and the tactile cues from our specific tactile sensor without any extra sensor calibrations. The robotic system is well controlled and adjusted based on the estimation model in real time. The main contributions and novelties of our work are listed as follows: 1) Explore a proprioceptive way for liquid volume estimation by developing an end-to-end predictive model with multi-modal convolutional networks, which achieve a high precision with an error of similar to 2 ml in the experimental validation. 2) Propose a multi-task learning architecture which comprehensively considers the losses from both classification and regression tasks, and comparatively evaluate the performance of each variant on the collected data and actual robotic platform. 3) Utilize the proprioceptive robotic system to accurately serve and control the requested volume of liquid, which is continuously flowing into a deformable container in real time. 4) Adaptively adjust the grasping plan to achieve more stable grasping and manipulation according to the real-time liquid volume prediction.

Visualization Adaptation models Propioception Magnetic liquids Grasping Containers Predictive models

Others

English

CPCI-S

Automation & Control Systems ; Computer Science ; Engineering ; Robotics

Automation & Control Systems ; Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic ; Robotics

WOS:000909405303133

IEEE

1.Department of Computer Science, The University of Hong Kong
2.Department of Biomedical Engineering, City University of Hong Kong
3.Department of Mechanical and Energy Engineering, Southern University of Science and Technology

Fan Zhu,Ruixing Jia,Lei Yang,et al. Visual-tactile Sensing for Real-time Liquid Volume Estimation in Grasping[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE,2022:12542-12549.