GAZE2REPORT: RADIOLOGY REPORT GENERATION VIA VISUAL-GAZE PROMPT TUNING OF LLMS.

Area of Science:

• Artificial Intelligence in Medical Imaging
• Radiology Report Generation
• Multimodal Data Fusion

Background:

• Current deep learning models for radiology reports lack physician-informed medical priors, leading to misalignments.
• Eye gaze data offers insights into radiologist attention, improving feature relevance and interpretability.
• Integrating eye gaze data is challenging due to multimodal fusion complexity and high acquisition costs, especially for inference.

Purpose of the Study:

• To develop a novel framework, Gaze2Report, for enhanced radiology report generation.
• To leverage eye gaze information for improved alignment between reports and disease manifestations.
• To enable AI models to function effectively without requiring eye gaze data during inference.

Main Methods:

• Gaze2Report utilizes a scanpath prediction module and a Graph Neural Network (GNN) to create joint visual-gaze tokens.
• These tokens, along with instruction and report tokens, form a multimodal prompt.
• The prompt is used to fine-tune LoRA layers of large language models (LLMs) for autoregressive report generation.

Main Results:

• The framework enhances radiology report quality through eye-gaze-guided visual learning.
• On-the-fly scanpath prediction allows the model to generate reports without direct gaze input during inference.
• Improved alignment between structured explanations and disease manifestations is achieved.

Conclusions:

• Gaze2Report effectively integrates eye gaze data into AI-driven radiology report generation.
• The proposed method addresses practical limitations of using gaze data in clinical settings.
• This approach offers a promising direction for more accurate and interpretable AI-assisted radiological diagnostics.