Abstract
Diabetic macular edema (DME) is a common cause of vision-threatening diseases. Multicolor image (MCI) enables the diagnosis of DME by providing multiple spectral images of fundus structures. However, the accuracy of existing machine learning methods is still low as they fail to exploit the characteristics of MCI. A multi-spectral vision transformer model with an attention fusion (Atfusion) module is proposed in this paper for DME classification. The transformer extracts the global features of the image using a self-attentive mechanism. In addition, a novel fusion technique - AtFusion module is created to efficiently fuse the multi-spectral features from both branches. We examine the empirical performance of the proposed algorithm on our in-house data sets. The classifier is able to predict the DME status of MCIs with accuracy of 0.951, sensitivity of 0.931, specificity of 0.953, and AUC of 0.933. The experimental results prove that the proposed methodology achieves relatively better performance than the state-of-the-art method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
The dataset during the current study is available from the corresponding author on reasonable request.
References
Acón D, Wu L (2018) Multimodal imaging in diabetic macular edema. The Asia-Pacific Journal of Ophthalmology 7(1):22–27
Carrington A, Manuel D, Fieguth P, Ramsay T, Osmani V, Wernly B, Bennett C, Hawken S, McInnes M, Magwood O, Sheikh Y, Holzinger A (2023) Deep roc analysis and auc as balanced average accuracy for improved classifier selection, audit and explanation. IEEE Trans Pattern Anal Mach Intell 45(1):329–341. https://doi.org/10.1109/TPAMI.2022.3145392
Chen C-F, Fan Q, Panda R (2021) Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103.14899
Chen C-F, Panda R, Fan Q (2021) Regionvit: Regional-to-local attention for vision transformers . arXiv:2106.02689
Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Jakob U, Neil H (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929
Fu J, Liu J, Tian H, Li Y, Bao Y, Fang Z, Lu H (2019) Dual attention network for scene segmentation, 3146–3154
Gao S, Cheng M, Zhao K, Zhang X, Yang M, Torr PH (2019) Res2net: A new multi-scale backbone architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence PP, 1–1 . https://doi.org/10.1109/TPAMI.2019.2938758
Gong R, Han R, Guo J, Liu W, Xu G (2021) Quantitative evaluation of hard exudates in diabetic macular edema by multicolor imaging and their associations with serum lipid levels. Acta Diabetologica 58 . https://doi.org/10.1007/s00592-021-01697-8
Guo Z, Li X, Huang H, Guo N, Li Q (2019) Deep learning-based image segmentation on multimodal medical imaging. IEEE Transactions on Radiation and Plasma Medical Sciences 162–169
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition, 770–778
Holló G, Aung T, Cantor LB, Aihara M (2020) Cystoid macular edema related to cataract surgery and topical prostaglandin analogs: mechanism, diagnosis, and management. Surv Ophthalmol 65(5):496–512
Horry M, Chakraborty S, Paul M, Ulhaq A, Pradhan B, Saha M, Shukla N (2020) Covid-19 detection through transfer learning using multimodal imaging data. IEEE Access PP, 1–1 . https://doi.org/10.1109/ACCESS.2020.3016780
Hu J, Shen L, Albanie S, Sun G, Vedaldi A (2018) Gather-excite: Exploiting feature context in convolutional neural networks. Advances in neural information processing systems 31
Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks, 7132–7141
Jemshi K, Gopi V, Niwas S I (2018) Development of an efficient algorithm for the detection of macular edema from optical coherence tomography images. International Journal of Computer Assisted Radiology and Surgery 13 . https://doi.org/10.1007/s11548-018-1795-6
Jin K, Pan X, You K, Wu J, Liu Z, Cao J, Lou L, Xu Y, Su Z, Yao K, Ye J (2020) Automatic detection of non-perfusion areas in diabetic macular edema from fundus fluorescein angiography for decision making using deep learning. Sci Rep 10:15138. https://doi.org/10.1038/s41598-020-71622-6
Kang EY-C, Yeung L, Lee Y-L, Wu C-H, Peng S-Y, Chen Y-P, Gao Q-Z, Lin C, Kuo C-F, Lai EC-C (2021) A multimodal imaging-based deep learning model for detecting treatment-requiring retinal vascular diseases: Model development and validation study. JMIR Med Inform 9:e28868. https://doi.org/10.2196/28868
Kavitha S, Thyagharajan Kandaswamy K (2017) Efficient DWT-based fusion techniques using genetic algorithm for optimal parameter estimation. Soft Comput 21(12):3307–3316. https://doi.org/10.1007/s00500-015-2009-6
Kermany D, Goldbaum M, Cai W, Valentim C, Liang H-Y, Baxter S, McKeown A, Yang G, Wu X, Yan F, Dong J, Prasadha M, Pei J, Ting M, Zhu J, Li C, Hewett S, Dong J, Ziyar I, Zhang K (2018) Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172:1122-1131.e9. https://doi.org/10.1016/j.cell.2018.02.010
Kornblau IS, El-Annan JF (2019) Adverse reactions to fluorescein angiography: a comprehensive review of the literature. Surv Ophthalmol 64(5):679–693
Lee J-Y, Martin-Bastida A, Murueta-Goyena A, Gabilondo I, Cuenca N, Piccini P, Jeon B (2022) Multimodal brain and retinal imaging of dopaminergic degeneration in parkinson disease. Nat Rev Neurol 18(4):203–220
Li F, Wang Y, Xu T, Dong L, Yan L, Jiang M, xuedian z, Jiang H, Wu Z, Zou H (2021) Deep learning-based automated detection for diabetic retinopathy and diabetic macular oedema in retinal fundus photographs. Eye 1–9 . https://doi.org/10.1038/s41433-021-01552-8
Li F, Wang Y, Xu T, Dong L, Yan L, Jiang M, Zhang X, Jiang H, Wu Z, Zou H (2021) Deep learning-based automated detection for diabetic retinopathy and diabetic macular oedema in retinal fundus photographs. Eye 1–9
Mnih V, Heess N, Graves A, Koray K (2014) Recurrent models of visual attention. Advances in neural information processing systems 27
Müller H, Mayrhofer M, Veen E-B, Holzinger A (2021) The ten commandments of ethical medical ai. Computer 54:119–123. https://doi.org/10.1109/MC.2021.3074263
Nielsen KB, Lautrup ML, Andersen JK, Savarimuthu TR, Grauslund J (2019) Deep learning-based algorithms in screening of diabetic retinopathy: A systematic review of diagnostic performance. Ophthalmology Retina 3(4):294–304
Qian X, Pei J, Zhang H, Xie X, Yan L, Han C, Gao X, Zhang H, Zheng W, Sun Q, Lu L, Shung K (2021) Prospective assessment of breast cancer risk from multimodal multiview ultrasound images via clinically applicable deep learning. Nature Biomedical Engineering 5:522–532. https://doi.org/10.1038/s41551-021-00711-2
Rs R, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2020) Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision 128 . https://doi.org/10.1007/s11263-019-01228-7
Saeedi P, Petersohn I, Salpea P, Malanda B, Karuranga S, Unwin N, Colagiuri S, Guariguata L, Motala AA, Ogurtsova K, Jonathan ES, Bright D, Williams R (2019) Global and regional diabetes prevalence estimates for 2019 and projections for 2030 and 2045: Results from the international diabetes federation diabetes atlas. Diabetes Res Clin Pract 157:107843
Sahlsten J, Jaskari J, Kivinen J, Turunen L, Jaanio E, Hietala K, Kaski K (2019) Deep learning fundus image analysis for diabetic retinopathy and macular edema grading. Sci Rep 9(1):1–11
Scanlon PH (2019) Update on screening for sight-threatening diabetic retinopathy. Ophthalmic Res 62(4):218–224
Srinivasan K, K K T (2015) Analysis of multi-modality brain images using machine learning techniques, 1482–1486
Sutong W, Yin Y, Wang D, Wang Y, Jin Y (2021) Interpretability-based multimodal convolutional neural networks for skin lesion diagnosis. IEEE Transactions on Cybernetics. https://doi.org/10.1109/TCYB.2021.3069920
Tang F, Wang X, Ran A-R, Chan CK, Ho M, Yip W, Young AL, Lok J, Szeto S, Chan J, Fanny Y, Raymond W, Ziqi T, Dawei Y, Danny SN, Chen LJ, Marten B, Victor C, Kenneth L, Lai TH, Tan GS, Ting DS, Huang H, Chen H, Ma JH, Tang S, Leng T, Kakavand S, Mannil SS, Chang RT, Liew G, Gopinath B, Lai TY, Pang CP, Scanlon PH, Wong TY, Tham CC, Chen H, Heng P-A, Cheung CY (2021) A multitask deep-learning system to classify diabetic macular edema for different optical coherence tomography devices: a multicenter analysis. Diabetes Care 44(9):2078–2088
Ting DSW, Cheung CY-L, Lim G, Tan GSW, Quang ND, Gan A, Hamzah H, Garcia-Franco R, San Yeo IY, Lee SY, Lee SY, Wong EYM, Sabanayagam C, Baskaran M, Ibrahim F, Tan NC, Finkelstein EA, Lamoureux EL, Wong IY, Bressler NM, Sivaprasad S, Varma R, Jonas JB, He MG, Cheng C-Y, Cheung GCM, Aung T, Hsu W, Lee ML, Wong TY (2017) Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA 318(22):2211–2223
Touvron H, Cord M, Douze M, Massa F, Sablayrolles A, Jégou H (2021) Training data-efficient image transformers & distillation through attention, 10347–10357 (PMLR, 2021)
Touvron H, Cord M, Sablayrolles A, Synnaeve G, Jégou H (2021) Going deeper with image transformers. arXiv preprint arXiv:2103.17239
Varadarajan A, Bavishi P, Ruamviboonsuk P, Chotcomwongse P, Venugopalan S, Narayanaswamy A, Cuadros J, Kanai K, Bresnick G, Tadarati M, Silpa-Archa S, Limwattanayingyong J, Nganthavee V, Ledsam J, Keane P, Corrado G, Peng L, Webster D (2020) Predicting optical coherence tomography-derived diabetic macular edema grades from fundus photographs using deep learning. Nat Commun 11:130. https://doi.org/10.1038/s41467-019-13922-8
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A, Kaiser L, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems 30
Wang Z, Keane PA, Chiang M, Cheung CY, Wong TY, Ting DSW (2020) Artificial intelligence and deep learning in ophthalmology. Artificial Intelligence in Medicine 1–34
Wilkinson C, Ferris FL, Klein RE, Lee PP, Agardh CD, Davis M, Dills D, Kampik A, Pararajasegaram R, Verdaguer JT (2003) Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110(9):1677–1682
Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: Convolutional block attention module, 3–19
Wu H, Xiao B, Codella N, Liu M, Dai X, Yuan L, Zhang L (2021) Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15808
Wu Q, Zhang B, Hu Y, Liu B, Cao D, Yang D, Peng Q, Zhong P, Zeng X, Xiao Y, C L, Y F, S F, M H, H C, X Y, H Y (2021) Detection of morphologic patterns of diabetic macular edema using a deep learning approach based on optical coherence tomography images. Retina (Philadelphia, Pa.) 41 (5), 1110
Wu Q, Zhang B, Hu Y, Liu B, Cao D, Yang D, Peng Q, Zhong P, Zeng X, Xiao Y, Li C, Fang Y, Feng S, Huang M, Cai H, Yang X, Yu H (2020) Detection of morphologic patterns of diabetic macular edema using a deep learning approach based on optical coherence tomography images. Retina (Philadelphia, Pa.) Publish Ahead of Print . https://doi.org/10.1097/IAE.0000000000002992
Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks, Vol. 2017-January, 5987–5995 . arXiv:1611.05431
Yuan L, Chen Y, Wang T, Yu W, Shi Y, Jiang Z, Tay FE, Feng J, Yan S (2021) Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986
Zagoruyko S, Komodakis N (2016) Wide Residual Networks. British Machine Vision Conference 2016, BMVC 2016, 87.1–87.12 . https://doi.org/10.5244/C.30.87. arXiv:1605.07146
Zhang Q, Liu Z, Li J, Liu G (2020) Identifying diabetic macular edema and other retinal diseases by optical coherence tomography image and multiscale deep learning. Diabetes, Metabolic Syndrome and Obesity: Targets and Therapy 13:4787
Zhou D, Kang B, Jin X, Yang L, Lian X, Jiang Z, Hou Q, Feng J (2021) Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103.11886
Acknowledgements
This work was supported by the Shandong Provincial Natural Science joint Foundation [Grant Number ZR2021MH237 and ZR2021LZL011], the National Natural Science Foundation of China [Grant Numbers 81871508 and 61773246], the Major Program of Shandong Province Natural Science Foundation [Grant Number ZR2018ZB0419], the Major Basic Research Program of Natural Science Foundation of Shandong Province [Grant Number ZR2019ZD04], and the Taishan Scholar Foundation of Shandong Province [Grant Number TSHW201502038].
Funding
The authors have not disclosed any funding.
Author information
Authors and Affiliations
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Ethical approval and informed consent
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
He, J., Song, J., Han, Z. et al. Multi-spectral transformer with attention fusion for diabetic macular edema classification in multicolor image. Soft Comput 28, 6117–6127 (2024). https://doi.org/10.1007/s00500-023-09417-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-023-09417-w