Global–local multi-stage temporal convolutional network for cataract surgery phase recognition

Fang，Lixin1,2; Mou，Lei2; Gu，Yuanyuan2,8; Hu，Yan3; Chen，Bang2; Chen，Xu4,5,6,7; Wang，Yang9; Liu，Jiang3; Zhao，Yitian2,8

2022-12-01

10.1186/s12938-022-01048-w

BioMedical Engineering Online   Impact Factor & Quartile

1475-925X

Background: Surgical video phase recognition is an essential technique in computer-assisted surgical systems for monitoring surgical procedures, which can assist surgeons in standardizing procedures and enhancing postsurgical assessment and indexing. However, the high similarity between the phases and temporal variations of cataract videos still poses the greatest challenge for video phase recognition. Methods: In this paper, we introduce a global–local multi-stage temporal convolutional network (GL-MSTCN) to explore the subtle differences between high similarity surgical phases and mitigate the temporal variations of surgical videos. The presented work consists of a triple-stream network (i.e., pupil stream, instrument stream, and video frame stream) and a multi-stage temporal convolutional network. The triple-stream network first detects the pupil and surgical instruments regions in the frame separately and then obtains the fine-grained semantic features of the video frames. The proposed multi-stage temporal convolutional network improves the surgical phase recognition performance by capturing longer time series features through dilated convolutional layers with varying receptive fields. Results: Our method is thoroughly validated on the CSVideo dataset with 32 cataract surgery videos and the public Cataract101 dataset with 101 cataract surgery videos, outperforming state-of-the-art approaches with 95.8% and 96.5% accuracy, respectively. Conclusions: The experimental results show that the use of global and local feature information can effectively enhance the model to explore fine-grained features and mitigate temporal and spatial variations, thus improving the surgical phase recognition performance of the proposed GL-MSTCN.

Cataract surgery videos Deep learning Surgical phase recognition Temporal convolutional networks

SCI

English

Others

Natural Science Foundation of Ningbo[202003N4039];Natural Science Foundation of Ningbo Municipality[202003N4039];

Engineering

Engineering, Biomedical

WOS:000892987300001

BMC

2-s2.0-85143117573

Scopus

1.College of Mechanical Engineering,Zhejiang University of Technology,Hangzhou,310014,China
2.Cixi Institute of Biomedical Engineering,Ningbo Institute of Materials Technology and Engineering,Chinese Academy of Sciences,Ningbo,China
3.Department of Computer Science and Engineering,Southern University of Science and Technology,Shenzhen,518055,China
4.Department of Ophthalmology,Shanghai Aier Eye Hospital,Shanghai,China
5.Department of Ophthalmology,Shanghai Aier Qingliang Eye Hospital,Shanghai,China
6.Aier Eye Hospital,Jinan University,Guangzhou,No. 601, Huangpu Road West,China
7.Aier School of Ophthalmology,Central South University Changsha,Changsha,Hunan,China
8.Zhejiang Engineering Research Center for Biomedical Materials,Cixi Institute of Biomedical Engineering,Ningbo Institute of Materials Technology and Engineering,Chinese Academy of Sciences,Ningbo,315300,China
9.Aerospace Information Research Institute,Chinese Academy of Sciences,Beijing,China

Fang，Lixin,Mou，Lei,Gu，Yuanyuan,等. Global–local multi-stage temporal convolutional network for cataract surgery phase recognition[J]. BioMedical Engineering Online,2022,21(1).

Fang，Lixin.,Mou，Lei.,Gu，Yuanyuan.,Hu，Yan.,Chen，Bang.,...&Zhao，Yitian.(2022).Global–local multi-stage temporal convolutional network for cataract surgery phase recognition.BioMedical Engineering Online,21(1).

Fang，Lixin,et al."Global–local multi-stage temporal convolutional network for cataract surgery phase recognition".BioMedical Engineering Online 21.1(2022).