Instrument-tissue Interaction Quintuple Detection in Surgery Videos

Lin，Wenjun1,2; Hu，Yan2; Hao，Luoying2,3; Zhou，Dan4; Yang，Mingming5; Fu，Huazhu6; Chui，Cheekong1; Liu，Jiang2

10.1007/978-3-031-16449-1_38

2022

25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)

0302-9743

1611-3349

978-3-031-16448-4

LECT NOTES ARTIF INT

13437 LNCS

399-409

SEP 18-22, 2022

null,Singapore,SINGAPORE

GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND

SPRINGER INTERNATIONAL PUBLISHING AG

Instrument-tissue interaction detection in surgical videos is a fundamental problem for surgical scene understanding which is of great significance to computer-assisted surgery. However, few works focus on this fine-grained surgical activity representation. In this paper, we propose to represent instrument-tissue interaction as ⟨ instrument bounding box, tissue bounding box, instrument class, tissue class, action class ⟩ quintuples. We present a novel quintuple detection network (QDNet) for the instrument-tissue interaction quintuple detection task in cataract surgery videos. Specifically, a spatiotemporal attention layer (STAL) is proposed to aggregate spatial and temporal information of the regions of interest between adjacent frames. We also propose a graph-based quintuple prediction layer (GQPL) to reason the relationship between instruments and tissues. Our method achieves an mAP of 42.24% on a cataract surgery video dataset, significantly outperforming other methods.

Instrument-tissue interaction quintuple detection Surgery video Surgical scene understanding

Others

English

CPCI-S

National Natural Science Foundation of China[8210072776] ; Guangdong Provincial Department of Education[2020ZDZX 3043] ; Guangdong Basic and Applied Basic Research Foundation[2021A1515012195] ; Shenzhen Natural Science Fund[JCYJ20200109140820699] ; AME Programmatic Fund[A20H4b0141] ; Stable Support Plan Program[20200925174052004]

Computer Science ; Imaging Science & Photographic Technology ; Radiology, Nuclear Medicine & Medical Imaging

Computer Science, Interdisciplinary Applications ; Imaging Science & Photographic Technology ; Radiology, Nuclear Medicine & Medical Imaging

WOS:000867568000038

2-s2.0-85139086379

Scopus

1.Department of Mechanical Engineering,National University of Singapore,Queenstown,Singapore
2.Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering,Southern University of Science and Technology,Shenzhen,China
3.School of Computer Science,University of Birmingham,Birmingham,United Kingdom
4.School of Ophthalmology and Optometry,Wenzhou Medical University,Wenzhou,China
5.Department of Ophthalmology,Shenzhen People’s Hospital,Shenzhen,China
6.Agency for Science,Technology and Research (A*STAR),Queenstown,Singapore

Lin，Wenjun,Hu，Yan,Hao，Luoying,et al. Instrument-tissue Interaction Quintuple Detection in Surgery Videos[C]. GEWERBESTRASSE 11, CHAM, CH-6330, SWITZERLAND:SPRINGER INTERNATIONAL PUBLISHING AG,2022:399-409.