Multi-modal fusion in thermal imaging and MRI for early cancer detection

Early detection of cancer relies on precise imaging that captures both structural and metabolic details, critical for identifying small yet significant tissue anomalies. Thermal imaging detects temperature changes linked to increased metabolic activity in cancerous tissues, while Magnetic Resonance Imaging (MRI) provides high-resolution soft tissue contrast. However, traditional single-modality imaging techniques can lack sufficient sensitivity or anatomical context when used alone. To address this, we propose a novel end-to-end unsupervised Multi-Mode Fused Recursive Neural Network (MMF-RvNN) framework modified to fuse thermal imaging with MRI, generating high-quality fused images that capture both thermal and MRI anatomical breast cancer information to enhance early-stage breast cancer detection. The breast cancer thermal and MRI images were subjected to a non-rigid registration process using anatomical landmarks to align both modalities accurately. The preprocessing steps include min-max normalization, bilinear interpolation, and Non-Local Means (NLM) methods are applied to normalize intensity values, adjust pixel size, and reduce noise in the scanning images. The multi-resolution analysis process involves using Static Curvelet Transform (SCLT) to decompose images into various frequency bands for feature extraction. Fusion techniques include the integration of SCLT and t-distributed Stochastic Neighbor Embedding (t-SNE) for low-frequency and high-frequency fusion, respectively. The novel MMF-RvNN generates fused output images by training a generator and evaluating their realism. Loss functions include content, style, and L1 losses to preserve texture and structural information. The result demonstrates the proposed model achieves a high rate of structural similarity index measure (SSIM) is 0.98, peak signal-to-noise ratio (PSNR) is 45.9 dB, and entropy (ENT) is 8.7 then decreasing the High-Frequency Error Norm (HFEN) by 0.019 and Normalized Mean Square Error (NMSE) by 0.006 and mutual information (MI) by 0.91. The MMF-RvNN achieves highly accurate detection the breast cancer tumors by providing a fused image that integrates MRI's soft-tissue detail with thermal imaging's metabolic insights. This method could lead to reduced radiotherapy planning time and improved target delineation, ultimately advancing clinical decision-making in oncology.