CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection

Nwoye，Chinedu Innocent1; Yu，Tong1; Sharma，Saurav1; Murali，Aditya1; Alapatt，Deepak1; Vardazaryan，Armine1,21; Yuan，Kun1,11; Hajek，Jonas2; Reiter，Wolfgang2; Yamlahi，Amine3; Smidt，Finn Henri3; Zou，Xiaoyang5; Zheng，Guoyan5; Oliveira，Bruno6,7,8; Torres，Helena R.6,7,8; Kondo，Satoshi9; Kasai，Satoshi10; Holm，Felix11; Özsoy，Ege11; Gui，Shuangchun12; Li，Han12; Raviteja，Sista13; Sathish，Rachana13; Poudel，Pranav16; Bhattarai，Binod14,15; Wang，Ziheng18; Rui，Guo18; Schellenberg，Melanie3,4; Vilaça，João L.6; Czempiel，Tobias11; Wang，Zhenkun12; Sheet，Debdoot13; Thapa，Shrawan Kumar17; Berniker，Max18; Godau，Patrick3,4; Morais，Pedro6; Regmi，Sudarshan17; Tran，Thuy Nuong3; Fonseca，Jaime8; Nölke，Jan Hinrich3,4; Lima，Estevão7; Vazquez，Eduard16; Maier-Hein，Lena3; Navab，Nassir11; Mascagni，Pietro19; Seeliger，Barbara1,20,21; Gonzalez，Cristians20,21; Mutter，Didier20,21; Padoy，Nicolas1,21

2023-10-01

Action detection CholecT50 Computer-assisted surgery Fine-grained activity recognition Surgical action triplet Tool localization Weak supervision

English

Horizon 2020[101002198];Science and Technology Commission of Shanghai Municipality[20511105205];Bundesministerium für Gesundheit[2520DAT0P1];

Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of ‹instrument, verb, target› triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.

10.1016/j.media.2023.102888

Medical Image Analysis

89

1361-8415

SCI

Others

2-s2.0-85164670806

Scopus

WOS:001047010000001

1361-8423

COMPUTER SCIENCE

1.ICube,University of Strasbourg,CNRS,France
2.Riwolink GmbH,Germany
3.Division of Intelligent Medical Systems (IMSY),German Cancer Research Center (DKFZ),Heidelberg,Germany
4.National Center for Tumor Diseases (NCT),Heidelberg,Germany
5.Institute of Medical Robotics,School of Biomedical Engineering,Shanghai Jiao Tong University,China
6.2Ai School of Technology,IPCA,Barcelos,Portugal
7.Life and Health Science Research Institute (ICVS),School of Medicine,University of Minho,Braga,Portugal
8.Algoritimi Center,School of Engineering,University of Minho,Guimeraes,Portugal
9.Muroran Institute of Technology,Japan
10.Niigata University of Health and Welfare,Japan
11.Technical University Munich,Germany
12.Southern University of Science and Technology,China
13.Indian Institute of Technology,Kharagpur,India
14.University College,London,United Kingdom
15.University of Aberdeen,United Kingdom
16.Redev Technology Ltd,United Kingdom
17.Nepal Applied Mathematics and Informatics Institute for research (NAAMII),Nepal
18.Intuitive Surgical,United States
19.Fondazione Policlinico Universitario Agostino Gemelli IRCCS,Rome,Italy
20.University Hospital of Strasbourg,France
21.IHU Strasbourg,France

Nwoye，Chinedu Innocent,Yu，Tong,Sharma，Saurav,等. CholecTriplet2022: Show me a tool and tell me the triplet — An endoscopic vision challenge for surgical action triplet detection. 2023-10-01.