MDF-Net for abnormality detection by fusing X-rays with clinical data | JoVE Visualize

Area of Science:

Medical Imaging and Artificial Intelligence
Radiology and Diagnostic Medicine
Computational Pathology

Background:

Current deep learning (DL) models for chest X-ray analysis achieve high performance using image data alone.
Radiologists emphasize the critical role of patient clinical information in accurate diagnosis and image interpretation.
Integrating diverse data modalities presents a challenge due to differing dimensional spaces.

Purpose of the Study:

To investigate the impact of incorporating patient clinical data on the performance of DL classifiers for disease localization in chest X-rays.
To propose and evaluate a novel multimodal DL architecture capable of processing both clinical and image data.
To enhance the accuracy and reliability of automated disease detection in thoracic imaging.

Main Methods:

Developed a novel DL architecture with two fusion methods for simultaneous processing of structured clinical data and unstructured chest X-ray images.
Introduced a spatialization strategy to facilitate multimodal learning within a Mask R-CNN framework, addressing dimensional differences.
Conducted extensive experiments using the MIMIC-Eye dataset, including MIMIC-CXR, MIMIC IV-ED, and REFLACX.

Main Results:

The proposed multimodal DL model achieved a 12% improvement in Average Precision for disease localization compared to a standard Mask R-CNN using only chest X-rays.
Incorporating patient clinical data significantly enhanced the performance of DL models for disease localization tasks.
Ablation studies confirmed the critical contribution of multimodal DL architectures and the inclusion of clinical data.

Conclusions:

Multimodal DL architectures integrating patient clinical data and chest X-rays offer superior performance for disease localization.
The proposed fusion methods and spatialization strategy effectively enable multimodal learning for improved diagnostic accuracy.
This research underscores the value of combining diverse data sources for more robust and reliable AI-driven medical image analysis.