MedFusionNet: A hybrid transformer-based multimodal deep learning framework for chronic disease prediction in women's...

Early detection of women's health conditions such as breast cancer, cervical cancer, and polycystic ovary syndrome (PCOS) remains a major challenge because these diseases often require analysing both visual and textual clinical information. Traditional deep learning systems usually depend upon one single source of input - the medical images or the clinical text which restricts them from understanding the full clinical picture. We propose MedFusionNet, a hybrid Transformer-based deep learning model that combines both medical images and clinical reports for more detailed information. The image processing stream introduces EfficientNetB0 and ViT to extract the detailed and global features of images, while in the text processing stream, a transformer based language model identifies the exact meaning of clinical notes. These two models are brought together with a cross-modal influence fusion mechanism that allows the model to learn how the visual patterns use the textual descriptions. After training on 18,434 image-text pairs, MedFusionNet shows good performance after its two-stage training process, learning the fusion and classification layers first, followed by the fine-tuning of all model components. The model achieved very high validation accuracies of 96.8% for breast cancer, 98.2% for cervical cancer, and 99.3% for PCOS detection with strong generalisation ability across diseases. This research further shows that combining image and text inputs will be able to make disease predictions early and accurately, which will make it easy to improve medical support and women's healthcare outcomes.