Eye-Gaze-Guided Vision Transformer for Rectifying Shortcut Learning.

Area of Science:

• Artificial Intelligence
• Medical Image Analysis
• Computer Vision

Background:

• Deep neural networks can learn spurious correlations, hindering generalizability and interpretability.
• Medical imaging analysis requires reliable, generalizable, and transparent models, often with limited data.
• Harmful shortcut learning is a significant challenge in developing effective AI for medical diagnostics.

Purpose of the Study:

• To propose a novel eye-gaze-guided vision transformer (EG-ViT) model for medical image analysis.
• To rectify harmful shortcut learning in deep neural networks using expert knowledge.
• To enhance the generalizability and interpretability of AI models in medical imaging.

Main Methods:

• Developed an EG-ViT model that integrates radiologist eye-gaze data to guide attention.
• Input masked image patches within radiologists' focus areas into the EG-ViT model.
• Implemented a residual connection to the last encoder layer to preserve patch interactions.

Main Results:

• The EG-ViT model effectively mitigated shortcut learning in medical imaging datasets.
• Demonstrated improved model interpretability compared to baseline methods.
• Showcased enhanced performance of large-scale vision transformers with limited samples by infusing domain knowledge.

Conclusions:

• The EG-ViT model successfully combines deep learning strengths with human expert prior knowledge.
• This approach rectifies harmful shortcut learning and improves AI model reliability in medical applications.
• The study opens new avenues for integrating human intelligence into AI paradigms for advanced diagnostics.