CholecTriplet2022: Show me a tool and tell me the triplet-An endoscopic vision challenge for surgical action triplet detection

Nwoye, Chinedu Innocent1; Yu, Tong1; Sharma, Saurav1; Murali, Aditya1; Alapatt, Deepak1; Vardazaryanau, Armine1,21; Yuan, Kun1,11; Hajek, Jonas2; Reiter, Wolfgang2; Yamlahi, Amine3; Smidt, Finn-Henri3; Zou, Xiaoyang5; Zheng, Guoyan5; Oliveira, Bruno6,7,8; Torres, Helena R.6,7,8; Kondo, Satoshi9; Kasai, Satoshi10; Holm, Felix11; Oezsoy, Ege11; Gui, Shuangchun12; Li, Han12; Raviteja, Sista13; Sathish, Rachana13; Poudel, Pranav16; Bhattarai, Binod14,15; Wang, Ziheng18; Rui, Guo18; Schellenberg, Melanie3,4; Vilaca, Joao L.6; Czempiel, Tobias11; Wang, Zhenkun; Sheet, Debdoot; Thapa, Shrawan Kumar; Berniker, Max; Godau, Patrick3,4; Morais, Pedro6; Regmi, Sudarshan17; Tran, Thuy Nuong3; Fonseca, Jaime8; Noelke, Jan-Hinrich3,4; Lima, Estevao7; Vazquez, Eduard16; Maier-Hein, Lena3; Navab, Nassir11; Mascagni, Pietro19; Seeliger, Barbara1,20,21; Gonzalez, Cristians20,21; Mutter, Didier20,21; Padoy, Nicolas1,21

2023-10-01

10.1016/j.media.2023.102888

MEDICAL IMAGE ANALYSIS   Impact Factor & Quartile

1361-8415

1361-8423

Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of & LANGBRAC;instrument,verb, target & rang; triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.

Action detection Tool localization Fine-grained activity recognition Surgical action triplet Weak supervision CholecT50 Computer-assisted surgery

SCI

English

Others

ANR["ANR-20-CHIA-0029-01","ANR-11-LABX-0004","ANR-16-CE33-0009","ANR-10-IAHU-02"] ; BPI France[DOS0180017/00] ; IHU Strasbourg["2021-AD011011638R2","2021-AD011011638R3"] ; Surgical Oncology Program of the National Center for Tumor Diseases(NCT) Heidelberg[20511105205] ; HELMHOLTZ IMAGING, a platform of the Helmholtz Information amp;Data Science Incubator, Germany[2520DAT0P1] ; Surgical Oncology Program of the National Center for Tumor Diseases (NCT) Heidelberg, Germany[101002198] ; Northern Portugal Regional Operational Programme (NORTE)["SFRH/BD/136721/2018","SFRH/BD/136670/2018"] ; FCT and FCT/MCTES["NORTE-01-0145-FEDER-000045","NORTE-01-0145-FEDER-000059"] ; CarlZeiss AG, Germany["UIDB/05549/2020","UIDP/05549/2020"] ; null[pdjh2023c21602]

Computer Science ; Engineering ; Radiology, Nuclear Medicine & Medical Imaging

Computer Science, Artificial Intelligence ; Computer Science, Interdisciplinary Applications ; Engineering, Biomedical ; Radiology, Nuclear Medicine & Medical Imaging

WOS:001047010000001

ELSEVIER

COMPUTER SCIENCE

Web of Science

1.Univ Strasbourg, CNRS, ICube, Strasbourg, France
2.Riwolink GmbH, Karlsruhe, Germany
3.German Canc Res Ctr, Div Intelligent Med Syst IMSY, Heidelberg, Germany
4.Natl Ctr Tumor Dis NCT, Heidelberg, Germany
5.Shanghai Jiao Tong Univ, Inst Med Robot, Sch Biomed Engn, Shanghai, Peoples R China
6.IPCA, 2Ai Sch Technol, Barcelos, Portugal
7.Univ Minho, Life & Hlth Sci Res Inst ICVS, Sch Med, Braga, Portugal
8.Univ Minho, Algoritimi Ctr, Sch Engn, Guimeraes, Portugal
9.Muroran Inst Technol, Muroran, Japan
10.Niigata Univ Hlth & Welf, Niigata, Japan
11.Tech Univ Munich, Munich, Germany
12.Southern Univ Sci & Technol, Shenzhen, Peoples R China
13.Indian Inst Technol, Kharagpur, India
14.UCL, London, England
15.Univ Aberdeen, Aberdeen, Scotland
16.Redev Technol Ltd, London, England
17.Nepal Appl Math & Informat Inst Res NAAMII, Suryabinayak, Nepal
18.Intuit Surg, Sunnyvale, CA USA
19.Fdn Policlin Univ Agostino Gemelli IRCCS, Rome, Italy
20.Univ Hosp Strasbourg, Strasbourg, France
21.IHU Strasbourg, Strasbourg, France

Nwoye, Chinedu Innocent,Yu, Tong,Sharma, Saurav,et al. CholecTriplet2022: Show me a tool and tell me the triplet-An endoscopic vision challenge for surgical action triplet detection[J]. MEDICAL IMAGE ANALYSIS,2023,89.

Nwoye, Chinedu Innocent.,Yu, Tong.,Sharma, Saurav.,Murali, Aditya.,Alapatt, Deepak.,...&Padoy, Nicolas.(2023).CholecTriplet2022: Show me a tool and tell me the triplet-An endoscopic vision challenge for surgical action triplet detection.MEDICAL IMAGE ANALYSIS,89.

Nwoye, Chinedu Innocent,et al."CholecTriplet2022: Show me a tool and tell me the triplet-An endoscopic vision challenge for surgical action triplet detection".MEDICAL IMAGE ANALYSIS 89(2023).