Deep metric learning with mirror attention and fine triplet loss for fundus image retrieval in ophthalmology

Fang，Jiansheng1; Zeng，Ming7; Zhang，Xiaoqing2; Liu，Hongbo3; Zhao，Yitian4; Zhang，Peng5; Yang，Hong5; Liu，Junling6; Miao，Hanpei2; Hu，Yan2; Liu，Jiang2

2023-02-01

10.1016/j.bspc.2022.104277

Biomedical Signal Processing and Control   Impact Factor & Quartile

1746-8094

1746-8108

Fundus image retrieval can help ophthalmologists make evidence-based medico-decision by providing similar cases. Its basic task is to learn highly discriminative visual descriptors from image space, in which lesion features are the main differentiating clue. Lesions in fundus images appear small in size, similar in textures, and scatter around vessels, such as microaneurysms and hemorrhages. Hence, although a single small lesion has a saliently visual manifestation, its discriminative information is hard to reserve in the last image descriptors. For fundus images, the optic disc of the left and right eyes are symmetric, and the macular area lies in the central axis from the vertical view. Based on such spatial structure and lesion characteristics, we present a novel deep metric learning framework equipped with mirror attention to enhance the discriminative features of small and scattering lesions and encode them into image descriptors. The mirror attention can give lesions high attention scores by capturing spatial dependency of vertical and horizontal views, especially the relations between lesions and vessels. Based on the mirror attention, we further propose a new fine triplet loss to confine distances of positive pairs by exploiting the learned relevant degrees of positive pairs in a self-supervised manner. The fine triplet loss can help detect the subtle differences of positive pairs to improve the ranking performance of hit items. To demonstrate the effectiveness of improving retrieval performance, we conduct comprehensive experiments on the largest fundus dataset of diabetic retinopathy (DR) detection and achieve the best precision compared to counterparts. The experiments show that our method produces significant performance improvements for fundus image retrieval, especially the ranking quality of DR grades containing microaneurysms and hemorrhages. Our proposed mirror attention can be applied to off-the-shelf backbones and trained efficiently in an end-to-end manner for other medical images to obtain highly discriminative image descriptors.

Attention Content-based image retrieval Deep metric learning Evidence-based medico-decision Ophthalmology Triplet loss

SCI

English

Corresponding

General Program of National Natural Science Foundation of China[82272086] ; Guangdong Provincial Department of Education[2020ZDZX3043]

Engineering

Engineering, Biomedical

WOS:000875715200006

ELSEVIER SCI LTD

2-s2.0-85139850192

Scopus

1.School of Computer Science and Technology,Harbin Institute of Technology,Harbin,China
2.Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering,Southern University of Science and Technology,ShenZhen,China
3.CVTE Research,Guangzhou,China
4.Cixi Institute of Biomedical Engineering,Chinese Academy of Sciences,Ningbo,China
5.Cyberspace Institute of Advanced Technology,Guangzhou University,Guangzhou,China
6.Guangdong Armed Police Hospital,Guangzhou,China
7.The University of Hong Kong,Hong Kong,Hong Kong

Fang，Jiansheng,Zeng，Ming,Zhang，Xiaoqing,et al. Deep metric learning with mirror attention and fine triplet loss for fundus image retrieval in ophthalmology[J]. Biomedical Signal Processing and Control,2023,80.

Fang，Jiansheng.,Zeng，Ming.,Zhang，Xiaoqing.,Liu，Hongbo.,Zhao，Yitian.,...&Liu，Jiang.(2023).Deep metric learning with mirror attention and fine triplet loss for fundus image retrieval in ophthalmology.Biomedical Signal Processing and Control,80.

Fang，Jiansheng,et al."Deep metric learning with mirror attention and fine triplet loss for fundus image retrieval in ophthalmology".Biomedical Signal Processing and Control 80(2023).