Eye-Guided Multimodal Fusion: Toward an Adaptive Learning Framework Using Explainable Artificial Intelligence

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Radiology Training

Background:

• Interpreting diagnostic imaging is challenging for novice radiologists due to a lack of structured guidance and expert feedback.
• Identifying clinically relevant features in medical images requires significant expertise and can be a difficult task.

Purpose of the Study:

• To develop and validate an Eye-Gaze Guided Multimodal Fusion framework to enhance learning and decision-making in medical image interpretation.
• To leverage expert eye-tracking data to improve the accuracy and interpretability of AI models in radiology.

Main Methods:

• Integrated chest X-ray (CXR) images with expert fixation maps to capture visual attention patterns.
• Utilized a shared backbone architecture for joint processing of image and gaze data, minimizing noise in fixation data.
• Employed Gradient-weighted Class Activation Mapping (Grad-CAM) for interpretability validation and assessed classification performance and explanation alignment.

Main Results:

• The Eye-Gaze Guided Multimodal Fusion framework demonstrated effectiveness in improving model reliability and interpretability.
• Evaluations confirmed the framework's robustness under gaze noise and alignment with expert annotations.
• The system successfully highlighted regions of interest (ROIs) critical for accurate diagnosis based on expert visual attention.

Conclusions:

• The proposed framework offers a promising pathway toward intelligent, human-centered AI systems for medical imaging.
• This approach supports both diagnostic accuracy and enhances medical training for radiologists.
• Integrating expert eye-tracking data provides valuable insights for AI-driven medical image analysis.