Abstract
Determining the necessity of resecting malignant polyps during colonoscopy screen is crucial for patient outcomes, yet challenging due to the time-consuming and costly nature of histopathology examination. While deep learning-based classification models have shown promise in achieving optical biopsy with endoscopic images, they often suffer from a lack of explainability. To overcome this limitation, we introduce EndoFinder, a content-based image retrieval framework to find the ‘digital twin’ polyp in the reference database given a newly detected polyp. The clinical semantics of the new polyp can be inferred referring to the matched ones. EndoFinder pioneers a polyp-aware image encoder that is pre-trained on a large polyp dataset in a self-supervised way, merging masked image modeling with contrastive learning. This results in a generic embedding space ready for different downstream clinical tasks based on image retrieval. We validate the framework on polyp re-identification and optical biopsy tasks, with extensive experiments demonstrating that EndoFinder not only achieves explainable diagnostics but also matches the performance of supervised classification models. EndoFinder’s reliance on image retrieval has the potential to support diverse downstream decision-making tasks during real-time colonoscopy procedures.
R. Yang and Y. Zhu—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Siegel, R.L., Miller, K.D., Fuchs, H.E., Jemal, A.: Cancer statistics, 2022. CA Cancer J. Clin. 72, 7–33 (2022). https://doi.org/10.3322/caac.21708
Siegel, R.L., Miller, K.D., Goding Sauer, A., et al.: Colorectal cancer statistics, 2020. CA Cancer J. Clin. 70, 145–164 (2020). https://doi.org/10.3322/caac.21601
Biller, L.H., Schrag, D.: Diagnosis and treatment of metastatic colorectal cancer: a review. JAMA 325, 669–685 (2021). https://doi.org/10.1001/jama.2021.0106
Chen, T., Kornblith, S., Norouzi, M., et al.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
Zhang, H., Cisse, M., Dauphin, Y.N., et al.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
El-Nouby, A., Neverova, N., Laptev, I., et al.: Training vision transformers for image retrieval. arXiv preprint arXiv:2102.05644 (2021)
Guan, A., Liu, L., Fu, X., et al.: Precision medical image hash retrieval by interpretability and feature fusion. Comput. Methods Programs Biomed. 222, 106945 (2022)
Wang, X., Du, Y., Yang, S., et al.: RetCCL: clustering-guided contrastive learning for whole-slide image retrieval. Med. Image Anal. 83, 102645 (2023)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al.: An image is worth \(16 \times 16\) words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
He, K., Chen, X., Xie, S., et al.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009 (2022)
Pizzi, E., Roy, S.D., Ravindra, S.N., Goyal, P., Douze, M.: A self-supervised descriptor for image copy detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14532–14542 (2022)
Hirsch, R., et al.: Self-supervised learning for endoscopic video analysis. In: Greenspan, H., et al. (eds.) MICCAI 2023. LNCS, vol. 14224, pp. 569–578. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43904-9_55
Shen, C., et al.: Forensic histopathological recognition via a context-aware mil network powered by self-supervised contrastive learning. In: Greenspan, H., et al. (eds.) MICCAI 2023. LNCS, vol. 14225, pp. 528–538. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43987-2_51
Grill, J.B., Strub, F., Altché, F., et al.: Bootstrap your own latent-a new approach to self-supervised learning. In: Advances in Neural Information Processing Systems, vol. 33, pp. 21271–21284 (2020)
Caron, M., Misra, I., Mairal, J., et al.: Unsupervised learning of visual features by contrasting cluster assignments. In: Advances in Neural Information Processing Systems, vol. 33, pp. 9912–9924 (2020)
Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Wu, Z., Xiong, Y., Yu, S.X., et al.: Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733–3742 (2018)
Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07737 (2017)
Öztürk, Ş: Class-driven content-based medical image retrieval using hash codes of deep features. Biomed. Signal Process. Control 68, 102601 (2021)
Liu, C., Ma, J., Tang, X., et al.: Deep hash learning for remote sensing image retrieval. IEEE Trans. Geosci. Remote Sens. 59(4), 3420–3443 (2020)
Li, T., Zhang, Z., Pei, L., et al.: HashFormer: vision transformer based deep hashing for image retrieval. IEEE Signal Process. Lett. 29, 827–831 (2022)
Chen, Y., Tang, Y., Huang, J., et al.: Multi-scale triplet hashing for medical image retrieval. Comput. Biol. Med. 155, 106633 (2023)
Chandran, S., Parker, F., Lontos, S., et al.: Can we ease the financial burden of colonoscopy? Using real-time endoscopic assessment of polyp histology to predict surveillance intervals. Int. Med. J. 45(12), 1293–1299 (2015)
van den Broek, F.J.C., Reitsma, J.B., Curvers, W.L., et al.: Systematic review of narrow-band imaging for the detection and differentiation of neoplastic and nonneoplastic lesions in the colon (with videos). Gastrointest. Endosc. 69(1), 124–135 (2009)
Ladabaum, U., Fioritto, A., Mitani, A., et al.: Real-time optical biopsy of colon polyps with narrow band imaging in community practice does not yet meet key thresholds for clinical decisions. Gastroenterology 144(1), 81–91 (2013)
Togashi, K., Osawa, H., Koinuma, K., et al.: A comparison of conventional endoscopy, chromoendoscopy, and the optimal-band imaging system for the differentiation of neoplastic and non-neoplastic colonic polyps. Gastrointest. Endosc. 69(3), 734–741 (2009)
Kuiper, T., Marsman, W.A., Jansen, J.M., et al.: Accuracy for optical diagnosis of small colorectal polyps in nonacademic settings. Clin. Gastroenterol. Hepatol. 10(9), 1016–1020 (2012)
Yamada, M., Shino, R., Kondo, H., et al.: Robust automated prediction of the revised Vienna classification in colonoscopy using deep learning: development and initial external validation. J. Gastroenterol. 57(11), 879–889 (2022)
Ribeiro, E., Uhl, A., Häfner, M.: Colonic polyp classification with convolutional neural networks. In: IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS), pp. 253–258. IEEE (2016)
Intrator, Y., Aizenberg, N., Livne, A., et al.: Self-supervised polyp re-identification in colonoscopy. arXiv preprint arXiv:2306.08591 (2023)
Chen, P.J., Lin, M.C., Lai, M.J., et al.: Accurate classification of diminutive colorectal polyps using computer-aided analysis. Gastroenterology 154(3), 568–575 (2018)
Björnsson, B., Borrebaeck, C., Elander, N., et al.: Digital twins to personalize medicine. Genome Med. 12, 1–4 (2020)
Krenzer, A., Heil, S., Fitting, D., et al.: Automated classification of polyps using deep learning architectures and few-shot learning. BMC Med. Imaging 23(1), 59 (2023)
Brearley, B.J., Bose, K.R., Senthil, K., et al.: KNN approaches by using ball tree searching algorithm with Minkowski distance function on smart grid data. Indian J. Comput. Sci. Eng 13(4), 1210–1226 (2022)
Wang, S., Zhu, Y., Luo, X., et al.: Knowledge extraction and distillation from large-scale image-text colonoscopy records leveraging large language and vision models. arXiv preprint arXiv:2310.11173 (2023)
Acknowledgement
This study was supported in part by the Shanghai Sailing Program (22YF1409300), International Science and Technology Cooperation Program under the 2023 Shanghai Action Plan for Science (23410710400) and National Natural Science Foundation of China (No. 62201263).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, R. et al. (2024). EndoFinder: Online Image Retrieval for Explainable Colorectal Polyp Diagnosis. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15010. Springer, Cham. https://doi.org/10.1007/978-3-031-72117-5_24
Download citation
DOI: https://doi.org/10.1007/978-3-031-72117-5_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72116-8
Online ISBN: 978-3-031-72117-5
eBook Packages: Computer ScienceComputer Science (R0)
