X-MedNet: An explainable multi-modal deep learning framework for brain tumor classification with clinical interpretability
Automatic classification of brain tumors from magnetic resonance imaging (MRI) remains a challenging task owing to tumor heterogeneity, class imbalance, and the limited interpretability of deep learning models. This study presents X-MedNet, an explainable multimodal deep learning framework designed for glioma classification by integrating complementary information from fluid-attenuated inversion recovery and contrast-enhanced T1-weighted MRI sequences through a two-channel input representation. Unlike conventional convolutional neural network-based approaches that primarily emphasize classification accuracy, X-MedNet combines multimodal feature learning, patient-level evaluation, and explainable artificial intelligence within a unified framework to promote a more realistic assessment of model performance. A modified ResNet18 architecture was adapted to process multimodal MRI data, while a class-weighted cross-entropy loss, data augmentation, and early stopping were employed to improve learning under class imbalance. Model interpretability was enhanced using gradient-weighted class activation mapping (Grad-CAM), enabling visualization of image regions contributing to the classification decision. The proposed framework was evaluated on the BraTS 2020 dataset using a patient-level train–validation–test split to reduce information leakage and better reflect real-world clinical scenarios. X-MedNet achieved an accuracy of 73.9%, precision of 85.6%, recall of 81.1%, an F1-score of 0.833, an area under the receiver operating characteristic curve of 0.707, an area under the precision–recall curve of 0.905, and a Matthews correlation coefficient of 0.241. Grad-CAM visualizations demonstrated that the network consistently focused on tumor-related regions, providing qualitative support for the interpretability of its predictions. Although further validation on external cohorts is required, the proposed framework demonstrates that integrating multimodal MRI, patient-level evaluation, and explainable artificial intelligence constitutes a promising approach toward reliable and transparent computer-aided glioma classification.
- Menze BH, Jakab A, Bauer S, et al. The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Trans Med Imaging. 2015;34(10):1993-2024. doi: 10.1109/TMI.2014.2377694
- Baid U, Ghodasara S, Mohan S, et al. The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification. arXiv. Preprint posted online 2021. doi: 10.48550/ARXIV.2107.02314
- Ker J, Wang L, Rao J, Lim T. Deep learning applications in medical image analysis. IEEE Access. 2018;6:9375-9389. doi: 10.1109/ACCESS.2017.2788044
- Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts HJWL. Artificial intelligence in radiology. Nat Rev Cancer. 2018;18(8):500-510. doi: 10.1038/s41568-018-0016-5
- Ribeiro MT, Singh S, Guestrin C. Why Should I Trust You? Explaining the Predictions of Any Classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016:1135-1144. doi: 10.1145/2939672.2939778
- Samek W, Montavon G, Lapuschkin S, Anders CJ, Müller KR. Explaining deep neural networks and beyond: A review of methods and applications. Proc IEEE. 2021;109(3):247-278. doi: 10.1109/JPROC.2021.3060483
- Holzinger A, Langs G, Denk H, Zatloukal K, Müller H. Causability and explainability of artificial intelligence in medicine. Wiley Interdiscip Rev Data Min Knowl Discov. 2019;9(4):e1312. doi: 10.1002/widm.1312
- Arrieta AB, Díaz-Rodríguez N, Del Ser J, et al. Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion. 2020;58:82-115. doi: 10.1016/j.inffus.2019.12.012
- Lundervold AS, Lundervold A. An overview of deep learning in medical imaging focusing on MRI. Z Med Phys. 2019;29(2):102-127. doi: 10.1016/j.zemedi.2018.11.002
- Erickson BJ, Korfiatis P, Akkus Z, Kline TL. Machine learning for medical imaging. Radiographics. 2017;37(2):505-515. doi: 10.1148/rg.2017160130
- Shen D, Wu G, Suk HI. Deep learning in medical image analysis. Annu Rev Biomed Eng. 2017;19(1):221-248. doi: 10.1146/annurev-bioeng-071516-044442
- Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44-56. doi: 10.1038/s41591-018-0300-7
- Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst. 2012;25:1097-1105.
- Dosovitskiy A, Beyer L, Kolesnikov A, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv. Preprint posted online 2020. doi: 10.48550/ARXIV.2010.11929
- Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016:2921-2929. doi: 10.1109/CVPR.2016.319
- de Vries BM, Zwezerijnen GJC, Burchell GL, van Velden FHP, Menke-van der Houven van Oordt CW, Boellaard R. Explainable artificial intelligence (XAI) in radiology and nuclear medicine: a literature review. Front Med. 2023;10:1180773. doi: 10.3389/fmed.2023.1180773
- Balakrishnan R, Hernández M del C V, Farrall AJ. Automatic segmentation of white matter hyperintensities from brain magnetic resonance images in the era of deep learning and big data—A systematic review. Comput Med Imaging Graph. 2021;88:101867. doi: 10.1016/j.compmedimag.2021.101867
- Lundberg SM, Lee SI. A unified approach to interpreting model predictions. Adv Neural Inf Process Syst. 2017;30:4765-4774.
- Bakas S, Akbari H, Sotiras A, et al. Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features. Sci Data. 2017;4(1):170117. doi: 10.1038/sdata.2017.117
- Rehman A, Naz S, Razzak MI, Akram F, Imran M. A deep learning-based framework for automatic brain tumors classification using transfer learning. Circ Syst Signal Process. 2020;39(2):757-775. doi: 10.1007/s00034-019-01246-3
- Bakas S, Reyes M, Jakab A, et al. Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge. arXiv. Preprint posted online 2018. doi: 10.48550/ARXIV.1811.02629
- Shorten C, Khoshgoftaar TM. A survey on image data augmentation for deep learning. J Big Data. 2019;6(1):60. doi: 10.1186/s40537-019-0197-0
- He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016:770-778. doi: 10.1109/CVPR.2016.90
- Rudie JD, Rauschecker AM, Bryan RN, Davatzikos C, Mohan S. Emerging Applications of Artificial Intelligence in Neuro-Oncology. Radiology. 2019;290(3):607-618. doi: 10.1148/radiol.2018181928
- Hatamizadeh A, Nath V, Tang Y, Yang D, Roth HR, Xu D. Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. In: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. Lecture Notes in Computer Science. Springer; 2022:272-284. doi: 10.1007/978-3-031-08999-2_22
- Simonyan K, Zisserman A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv. Preprint posted online 2014. doi: 10.48550/ARXIV.1409.1556
- Kingma DP, Ba J. Adam: A Method for Stochastic Optimization. arXiv. Preprint posted online 2014. doi: 10.48550/ARXIV.1412.6980
- Bottou L. Stochastic gradient descent tricks. In: Montavon G, Orr GB, Müller KR, eds. Neural Networks: Tricks of the Trade. 2nd ed. Springer; 2012:421-436. doi: 10.1007/978-3-642-35289-8_25
- Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N Engl J Med. 2019;380(14):1347-1358. doi: 10.1056/NEJMra1814259
- Fawcett T. An introduction to ROC analysis. Pattern Recognit Lett. 2006;27(8):861-874. doi: 10.1016/j.patrec.2005.10.010
- Chicco D, Jurman G. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics. 2020;21(1):6. doi: 10.1186/s12864-019-6413-7
- Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2017:618-626. doi: 10.1109/ICCV.2017.74
- Isensee F, Jaeger PF, Kohl SAA, Petersen J, Maier-Hein KH. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods. 2021;18(2):203-211. doi: 10.1038/s41592-020-01008-z
- Işın A, Direkoğlu C, Şah M. Review of MRI-based brain tumor image segmentation using deep learning methods. Procedia Comput Sci. 2016;102:317-324. doi: 10.1016/j.procs.2016.09.407
- Tjoa E, Guan C. A survey on explainable artificial intelligence (XAI): toward medical XAI. IEEE Trans Neural Netw Learn Syst. 2021;32(11):4793-4813. doi: 10.1109/TNNLS.2020.3027314
- Budhkar A, Song Q, Su J, Zhang X. Demystifying the black box: A survey on explainable artificial intelligence (XAI) in bioinformatics. Comput Struct Biotechnol J. 2025;27:346-359. doi: 10.1016/j.csbj.2024.12.027
- Rajpurkar P, Chen E, Banerjee O, Topol EJ. AI in health and medicine. Nat Med. 2022;28(1):31-38. doi: 10.1038/s41591-021-01614-0
- Sajjad M, Khan S, Muhammad K, Wu W, Ullah A, Baik SW. Multi-grade brain tumor classification using deep CNN with extensive data augmentation. J Comput Sci. 2019;30:174-182. doi: 10.1016/j.jocs.2018.12.003
- Cheng J, Huang W, Cao S, et al. Enhanced performance of brain tumor classification via tumor region augmentation and partition. PLoS ONE. 2015;10(10):e0140381. doi: 10.1371/journal.pone.0140381
- Amin J, Sharif M, Gul N, Yasmin M, Shad SA. Brain tumor classification based on DWT fusion of MRI sequences using convolutional neural network. Pattern Recognit Lett. 2020;129:115-122. doi: 10.1016/j.patrec.2019.11.016