基于多模态脑影像的有监督癫痫识别实现无监督病灶定位

    癫痫是神经元异常放电的神经系统疾病，发作时甚至可能威胁到人们的生命。神经外科手术是药物难治性癫痫患者治愈的希望，而神经外科手术需要精准地评估致痫病灶的性质、位置以及边界。比起电生理等其他成像模态，神经影像学在确定癫痫病灶的大小和范围方面具有重要的优势，这对术前诊断至关重要。磁共振影像具有无辐射、多样化对比度以及高空间分辨率的优点，因此在临床中广泛使用。然而，目前对磁共振影像的诊断存在不足之处。具体而言，首先是依靠临床经验进行诊断的方式具有高度的主观性，准确的检验依赖于阅片医生的专业性；其二，诊断需要多种模态互相弥补参考，医生需要遍览所有模态与切片的影像，工作量过大且容易因视觉疲劳出现纰漏；其三，部分致痫病灶由于体积微小或是在成像上与周围正常组织对比度差异不大，从而极难凭肉眼分辨出来，因此造成漏诊。

    深度学习在近十年的医学影像分析中发挥着越来越重要的作用，也催生了许多神经影像学临床应用的新进展。然而，深度学习在医学影像中的应用有其局限性：其一，深度学习是由数据驱动的，对数据的数量与质量有着极高的要求；其二，医学影像体素值较多，信息更为稠密，往往需要有详细的标签来辅助深度学习判断，然而医学图像领域的详细标签标注需要专业的医生耗时一层层勾画来完成，极难获得。针对以上问题，本课题提出了一种在小数据集上通过有监督地融合多种模态的癫痫分类模型来实现无监督地定位局灶性皮质发育不良病灶的方法。其中分类模型由三个主要模块组成：三维块切分模块、三维空间和通道卷积模块以及全连接特征融合分类模块；T1模态在特征图层面融合而DTI模态在特征提取层面融合；癫痫病灶的定位是由层级梯度加权类激活图完成的。在临床上局灶皮质发育不良癫痫数据集和开源的年龄匹配正常数据集上的实验结果表明，本课题设计的多模态分类模型参数较少，在小样本癫痫数据集上分类能够与多参数的卷积神经网络媲美，癫痫识别准确率能够在参数最少的情况下达到90%；定位结果与手术切除后经病理验证为皮质发育不良的区域匹配良好，磁共振阴性的病灶定位灵敏率能够达到72.7%，与有监督的机器学习分割技术相当。最后，通过依次消融影像模态得到的测试集准确率比较发现，DTI模态中的局部弥散均匀性图对癫痫分类最为重要。

[1] SONG P, LIU Y, YU X, et al. Prevalence of epilepsy in China between 1990 and 2015: A systematic review and meta–analysis[J]. Journal of Global Health, 2017, 7(2): 20706-20716.
[2] FIEST K M, SAURO K M, WIEBE S, et al. Prevalence and incidence of epilepsy: A systematic review and meta-analysis of international studies[J]. Neurology, 2017, 88(3): 296-303.
[3] BERG A T. Defining intractable epilepsy[J]. Advances in Neurology, 2006, 97: 5-10.
[4] ZHAO X, ZHOU Z, ZHU W, et al. Role of conventional magnetic resonance imaging in the screening of epilepsy with structural abnormalities: A pictorial essay[J]. American Journal of Nuclear Medicine and Molecular Imaging, 2017, 7(3): 126-137.
[5] FISHER R S, ACEVEDO C, ARZIMANOGLOU A, et al. Ilae official report: A practicalclinical definition of epilepsy[J]. Epilepsia, 2014, 55(4): 475-482.
[6] PASQUIER B, PÉOC’H M, FABRE-BOCQUENTIN B, et al. Surgical pathology of drugresistant partial epilepsy. a 10-year-experience with a series of 327 consecutive resections[J].Epileptic Disorders, 2002, 4(2): 99-119.
[7] WANG Z I, ALEXOPOULOS A V, JONES S E, et al. The pathology of magnetic-resonanceimaging-negative epilepsy[J]. Modern Pathology, 2013, 26(8): 1051-1058.
[8] SEVERINO M, GERALDO A F, UTZ N, et al. Definitions and classification of malformations of cortical development: Practical guidelines[J]. Brain, 2020, 143(10): 2874-2894.
[9] GIEDD J N. Structural magnetic resonance imaging of the adolescent brain[J]. Annals of The New York Academy of Sciences, 2004, 1021(1): 77-85.
[10] LITJENS G, KOOI T, BEJNORDI B E, et al. A survey on deep learning in medical image analysis[J]. Medical Image Analysis, 2017, 42: 60-88.
[11] YUAN J, RAN X, LIU K, et al. Machine learning applications on neuroimaging for diagnosis and prognosis of epilepsy: A review[J]. Journal of Neuroscience Methods, 2021: 109441-109455.
[12] POMINOVA M, ARTEMOV A, SHARAEV M, et al. Voxelwise 3d convolutional and recurrent neural networks for epilepsy and depression diagnostics from structural and functional MRI data[C]//2018 IEEE International Conference on Data Mining Workshops (ICDMW). IEEE, 2018:299-307.
[13] ALAVERDYAN Z, JUNG J, BOUET R, et al. Regularized siamese neural network for unsupervised outlier detection on brain multiparametric magnetic resonance imaging: Applicationto epilepsy lesion screening[J]. Medical Image Analysis, 2020, 60: 101618-101631.
[14] CARMO D, SILVA B, YASUDA C, et al. Hippocampus segmentation on epilepsy and Alzheimer's disease studies with multiple convolutional neural networks[J]. Heliyon, 2021, 7(2): e06226.
[15] EL AZAMI M, HAMMERS A, JUNG J, et al. Detection of lesions underlying intractable epilepsy on t1-weighted mri as an outlier detection problem[J]. PloS One, 2016, 11(9):e0161498.
[16] MUNSELL B, WU G, FRIDRIKSSON J, et al. Relationship between neuronal network architecture and naming performance in temporal lobe epilepsy: A connectome based approach using machine learning[J]. Brain and Language, 2019, 193: 45-57.
[17] GLEICHGERRCHT E, MUNSELL B, BHATIA S, et al. Deep learning applied to whole-brain connectome to determine seizure control after epilepsy surgery[J]. Epilepsia, 2018, 59 (9): 1643-1654.
[18] MEMARIAN N, KIM S, DEWAR S, et al. Multimodal data and machine learning for surgery outcome prediction in complicated cases of mesial temporal lobe epilepsy[J]. Computers in Biology And Medicine, 2015, 64: 67-78.
[19] BERNHARDT B C, HONG S J, BERNASCONI A, et al. Magnetic resonance imaging pattern learning in temporal lobe epilepsy: Classification and prognostics[J]. Annals of Neurology, 2015, 77(3): 436-446.
[20] SAMSON K. ‘deep learning’model using artificial intelligence predicts surgical success in intractable temporal lobe epilepsy[J]. Neurology Today, 2018, 18(23): 50-55.
[21] CANTOR-RIVERA D, KHAN A R, GOUBRAN M, et al. Detection of temporal lobe epilepsy using support vector machines in multi-parametric quantitative mr imaging[J]. Computerized Medical Imaging and Graphics, 2015, 41: 14-28.
[22] WANG J, LI Y, WANG Y, et al. Multimodal data and machine learning for detecting specific biomarkers in pediatric epilepsy patients with generalized tonic-clonic seizures[J]. Frontiers in Neurology, 2018, 9: 1038-1052.
[23] JIANG H Y, LIU R N, GAO F F, et al. Hemisphere symmetry feature based on tensor space and recognition of epilepsy[J]. Journal of Northeastern University (Natural Science), 2017, 38(7): 923-927.
[24] DEL GAIZO J, MOFRAD N, JENSEN J H, et al. Using machine learning to classify temporal lobe epilepsy based on diffusion mri[J]. Brain And Behavior, 2017, 7(10): e00801.
[25] LIEDLGRUBER M, BUTZ K, HÖLLER Y, et al. Can spharm-based features from automated or manually segmented hippocampi distinguish between mci and tle?[C]//Scandinavian Conference on Image Analysis. Springer, 2019: 465-476.
[26] HÖLLER Y, BUTZ K H, THOMSCHEWSKI A C, et al. Prediction of cognitive decline in temporal lobe epilepsy and mild cognitive impairment by eeg, mri, and neuropsychology[J]. Computational Intelligence And Neuroscience, 2020, 2020.
[27] KANG L, CHEN J, HUANG J, et al. Identifying epilepsy based on machine-learning technique with diffusion kurtosis tensor[J]. CNS Neuroscience & Therapeutics, 2021: 354-363.
[28] TORLAY L, PERRONE-BERTOLOTTI M, THOMAS E, et al. Machine learning–xgboost analysis of language networks to classify patients with epilepsy[J]. Brain Informatics, 2017, 4(3): 159-169.
[29] GHAZI N, SOLTANIAN-ZADEH H. Structural connectivity of temporal lobe structures detects temporal lobe epilepsy[C]//2016 23rd Iranian Conference on Biomedical Engineering and 2016 1st International Iranian Conference on Biomedical Engineering (ICBME). IEEE, 2016: 30-34.
[30] YAN M, LIU L, CHEN S, et al. A deep learning method for prediction of benign epilepsy with centrotemporal spikes[C]//ISBRA. 2018: 253-258.
[31] ITO Y, FUKUDA M, MATSUZAWA H, et al. Deep learning-based diagnosis of temporal lobe epilepsy associated with hippocampal sclerosis: An mri study[J]. Epilepsy Research, 2021, 178: 106815-106822.
[32] GLEICHGERRCHT E, MUNSELL B, KELLER S, et al. Radiological identification of temporal lobe epilepsy using artificial intelligence: A feasibility study[J]. Brain Communications, 2021.
[33] JIANG H, GAO F, DUAN X, et al. Transfer learning and fusion model for classification of epileptic pet images[M]//Innovation in Medicine and Healthcare Systems, and Multimedia. Springer, 2019: 71-79.
[34] EL AZAMI M, HAMMERS A, COSTES N, et al. Computer aided diagnosis of intractable epilepsy with mri imaging based on textural information[C]//2013 International Workshop on Pattern Recognition in Neuroimaging. IEEE, 2013: 90-93.
[35] AHMED B, THESEN T, BLACKMON K, et al. Hierarchical conditional random fields for outlier detection: An application to detecting epileptogenic cortical malformations[C]// International Conference on Machine Learning. 2014: 1080-1088.
[36] GILL R S, HONG S J, FADAIE F, et al. Deep convolutional networks for automated detection of epileptogenic brain malformations[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2018: 490-497.
[37] DEV K, JOGI P S, NIYAS S, et al. Automatic detection and localization of focal cortical dysplasia lesions in mri using fully convolutional neural network[J]. Biomed. Signal Process. Control., 2019, 52: 218-225.
[38] AMINPOUR A, EBRAHIMI M, WIDJAJA E. Lesion localization in paediatric epilepsy using patch-based convolutional neural network[C]//International Conference on Image Analysis and Recognition. Springer, 2020: 216-227.
[39] HOUSE P M, KOPELYAN M, BRANIEWSKA N, et al. Automated detection and segmentation of focal cortical dysplasias (fcds) with artificial intelligence: Presentation of a novel convolutional neural network and its prospective clinical validation[J]. Epilepsy Research, 2021, 172: 106594-106603.
[40] ACHARYA U R, HAGIWARA Y, DESHPANDE S N, et al. Characterization of focal eeg signals: A review[J]. Future Generation Computer Systems, 2019, 91: 290-299.
[41] BOONYAKITANONT P, LEK-UTHAI A, CHOMTHO K, et al. A review of feature extraction and performance evaluation in epileptic seizure detection using eeg[J]. Biomedical Signal Processing and Control, 2020, 57: 101702-101729.
[42] YASAKA K, ABE O. Deep learning and artificial intelligence in radiology: Current applications and future directions[J]. PLoS Medicine, 2018, 15(11): e1002707.
[43] ABBASI B, GOLDENHOLZ D M. Machine learning applications in epilepsy[J]. Epilepsia, 2019, 60(10): 2037-2047.
[44] PEREIRA F, MITCHELL T M, BOTVINICK M. Machine learning classifiers and fmri: A tutorial overview[J]. NeuroImage, 2009, 45: s199-s209.
[45] RAMACHANDRAM D, TAYLOR G W. Deep multimodal learning: A survey on recent advances and trends[J]. IEEE Signal Processing Magazine, 2017, 34(6): 96-108.
[46] CHANG K, BAI H X, ZHOU H, et al. Residual convolutional neural network for the determination of idh status in low-and high-grade gliomas from mr imaging[J]. Clinical Cancer Research, 2018, 24(5): 1073-1081.
[47] GUO Z, LI X, HUANG H, et al. Deep learning-based image segmentation on multimodal medical imaging[J]. IEEE Transactions on Radiation and Plasma Medical Sciences, 2019, 3(2): 162-169.
[48] LIU S, LIU S, CAI W, et al. Multimodal neuroimaging feature learning for multiclass diagnosis of Alzheimer's disease[J]. IEEE Transactions on Biomedical Engineering, 2014, 62(4): 1132-1140.
[49] SUK H I, LEE S W, SHEN D, et al. Hierarchical feature representation and multimodal fusion with deep learning for ad/MCI diagnosis[J]. NeuroImage, 2014, 101: 569-582.
[50] KIM J, LEE B. Identification of Alzheimer's disease and mild cognitive impairment using multimodal sparse hierarchical extreme learning machine[J]. Human Brain Mapping, 2018, 39(9):3728-3741.
[51] FENG C, ELAZAB A, YANG P, et al. Deep learning framework for alzheimer’s disease diagnosis via 3d-CNN and fsbi-lstm[J]. IEEE Access, 2019, 7: 63605-63618.
[52] HAO X, BAO Y, GUO Y, et al. Multi-modal neuroimaging feature selection with consistent metric constraint for diagnosis of Alzheimer's disease[J]. Medical Image Analysis, 2020, 60:101625-101637.
[53] LIANG S, ZHANG R, LIANG D, et al. Multimodal 3d densenet for idh genotype prediction in gliomas[J]. Genes, 2018, 9(8): 382-398.
[54] GUO X, WANG J, WANG X, et al. Diagnosing autism spectrum disorder in children using conventional MRI and apparent diffusion coefficient based deep learning algorithms[J]. European Radiology, 2022, 32(2): 761-770.
[55] NEUBAUER T, WIMMER M, BERG A, et al. Soft tissue sarcoma co-segmentation in combined mri and pet/ct data[M]//Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures. Springer, 2020: 97-105.
[56] RAJALINGAM B, PRIYA R. Multimodal medical image fusion based on deep learning neural network for clinical treatment analysis[J]. International Journal of ChemTech Research, 2018,11(06): 160-176.
[57] MUHAMMAD G, HOSSAIN M S. Covid-19 and non-covid-19 classification using multi-layers fusion from lung ultrasound images[J]. Information Fusion, 2021, 72: 80-88.
[58] HE S, PEREIRA D, PEREZ J D, et al. Multi-channel attention-fusion neural network for brain age estimation: Accuracy, generality, and interpretation with 16,705 healthy MRIs across lifespan[J]. Medical Image Analysis, 2021, 72: 102091-102107.
[59] GUO W, WANG J, WANG S. Deep multimodal representation learning: A survey[J]. IEEE Access, 2019, 7: 63373-63394.
[60] LIU S, WANG Y S, ZHANG Q, et al. Chinese color nest project: An accelerated longitudinal brain-mind cohort[J]. Developmental Cognitive Neuroscience, 2021, 52: 101020-101034.
[61] MCAULIFFE M J, LALONDE F M, MCGARRY D, et al. Medical image processing, analysis and visualization in clinical research[C]//Proceedings 14th IEEE Symposium on Computer-Based Medical Systems. CBMS 2001. IEEE, 2001: 381-386.
[62] SMITH S M. Bet: Brain extraction tool[J]. FMRIB TR00SMS2b, Oxford Centre for Functional Magnetic Resonance Imaging of the Brain), Department of Clinical Neurology, Oxford University, John Radcliffe Hospital, Headington, UK, 2000.
[63] JENKINSON M, SMITH S. A global optimisation method for robust affine registration of brain images[J]. Medical Image Analysis, 2001, 5(2): 143-156.
[64] CUI Z, ZHONG S, XU P, et al. Panda: a pipeline toolbox for analyzing brain diffusion images[J]. Frontiers in Human Neuroscience, 2013, 7: 42-57.
[65] RUMELHART D E, HINTON G E, WILLIAMS R J. Learning internal representations by error propagation[R]. California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
[66] SRIVASTAVA N, HINTON G E, KRIZHEVSKY A, et al. Dropout: A simple way to prevent neural networks from overfitting[J]. J. Mach. Learn. Res., 2014, 15: 1929-1958.
[67] IOFFE S, SZEGEDY C. Batch normalization: Accelerating deep network training by reducing internal covariate shift[C]//ICML. 2015: 448-456.
[68] MANASSI M, SAYIM B, HERZOG M H. When crowding of crowding leads to uncrowding[J]. Journal of Vision, 2013, 13(13): 10-10.
[69] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778.
[70] TOLSTIKHIN I, HOULSBY N, KOLESNIKOV A, et al. Mlp-mixer: An all-mlp architecture for vision[J]. ArXiv Preprint ArXiv:2105.01601, 2021.
[71] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words:Transformers for image recognition at scale[J]. ArXiv Preprint ArXiv:2010.11929, 2020.
[72] DAIR.AI. Ml visuals[EB/OL]. https://github.com/dair-ai/ml-visuals Accessed January 15, 2021.
[73] SONE D, BEHESHTI I. Clinical application of machine learning models for brain imaging in epilepsy: A review[J]. Frontiers in Neuroscience, 2021, 15: 761-774.
[74] ZHOU B, KHOSLA A, LAPEDRIZA A, et al. Learning deep features for discriminative localization[C]//Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition. 2016: 2921-2929.
[75] WANG H, WANG Z, DU M, et al. Score-cam: Score-weighted visual explanations for convolutional neural networks[C]//Proceedings of The IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020: 24-25.
[76] SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-cam: Visual explanations from deep networks via gradient-based localization[C]//Proceedings of The IEEE International Conference on Computer Vision. 2017: 618-626.
[77] CHATTOPADHYAY A, SARKAR A, HOWLADER P, et al. Grad-cam++: Improved visual explanations for deep convolutional networks[J]. ArXiv Preprint ArXiv:1710.11063, 2017.
[78] JIANG P T, ZHANG C B, HOU Q, et al. Layercam: Exploring hierarchical class activation maps for localization[J]. IEEE Transactions on Image Processing, 2021, 30: 5875-5888.
[79] ZHAO Y, AHMED B, THESEN T, et al. A non-parametric approach to detect epileptogenic lesions using restricted Boltzmann machines[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016: 373-382.
[80] WAGSTYL K, ADLER S, PIMPEL B, et al. Planning stereoelectroencephalography usingautomated lesion detection: Retrospective feasibility study[J]. Epilepsia, 2020, 61(7): 1406-1416.
[81] WIDJAJA E, GEIBPRASERT S, OTSUBO H, et al. Diffusion tensor imaging assessment of the epileptogenic zone in children with localization-related epilepsy[J]. American Journal of Neuroradiology, 2011, 32(10): 1789-1794.
[82] WINSTON G P, MICALLEF C, SYMMS M R, et al. Advanced diffusion imaging sequences could aid assessing patients with focal cortical dysplasia and epilepsy[J]. Epilepsy Research, 2014, 108(2): 336-339.