Relationship Oriented Semantic Scene Understanding for Daily Manipulation Tasks

Chao Tang1; Jingwen Yu2; Weinan Chen3; Bingyi Xia1; Hong Zhang1

10.1109/IROS47612.2022.9981960

2022

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

2153-0858

978-1-6654-7928-8

2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

9926-9933

23-27 Oct. 2022

Kyoto, Japan

345 E 47TH ST, NEW YORK, NY 10017 USA

IEEE

Assistive robot systems have been developed to help people accomplish daily manipulation tasks especially for those with disabilities, where scene understanding plays a crucial role in enabling robots to interpret the surroundings and behave accordingly. Most of the current systems approach scene understanding without considering the functional dependencies between objects. However, it is only valuable to interact with some objects when their function-relevant counterparts are considered. In this paper, we augment an assistive robotic arm system with an end-to-end semantic relationship reasoning model. It incorporates functional relationships between pairs of objects for semantic scene understanding. To ensure good generalization to unseen objects and relationships, the model works in a category-agnostic manner. We evaluate our design and three baseline methods on a self-collected benchmark with two levels of difficulty. To further demonstrate the effectiveness, the model is integrated with a symbolic planner for goaloriented, multi-step manipulation task on a real-world assistive robotic arm platform.

Visualization Semantics Pipelines Benchmark testing Manipulators Assistive robots Cognition

First

English

CPCI-S

National Natural Science Foundation of China[62103179] ; China Postdoctoral Science Foundation[2021M701576] ; Leading Talents Program of Guangdong Province[2016LJ06G498]

Automation & Control Systems ; Computer Science ; Engineering ; Robotics

Automation & Control Systems ; Computer Science, Artificial Intelligence ; Engineering, Electrical & Electronic ; Robotics

WOS:000909405302042

IEEE

1.Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China
2.Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China
3.Biomimetic and Intelligent Robotics Lab (BIRL), Guangdong University of Technology, Guangzhou, China

Chao Tang,Jingwen Yu,Weinan Chen,et al. Relationship Oriented Semantic Scene Understanding for Daily Manipulation Tasks[C]. 345 E 47TH ST, NEW YORK, NY 10017 USA:IEEE,2022:9926-9933.