Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-Ray Report Generation

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Natural Language Processing

Background:

• Large language models (LLMs) have advanced X-ray report generation but often overlook crucial visual details.
• Existing models produce linguistically sound reports that may lack sufficient clinical disease information.
• Effective integration of visual data and historical context is needed for superior medical report generation.

Purpose of the Study:

• To develop an associative memory-enhanced model for X-ray report generation.
• To improve the descriptive accuracy of medical reports by better utilizing visual information.
• To mimic the comprehensive report writing process of professional radiologists.

Main Methods:

• Utilized a classification model with activation maps to identify disease-related visual regions and learn disease query tokens.
• Employed a visual Hopfield network for memory association of disease tokens.
• Integrated a report Hopfield network to retrieve historical report information for enhanced generation.
• Leveraged a large language model for final report synthesis.

Main Results:

• Achieved state-of-the-art performance on benchmark datasets: IU X-ray, MIMIC-CXR, and Chexpert Plus.
• Demonstrated improved ability to describe key diseases compared to previous methods.
• Generated high-quality medical reports by effectively combining visual and textual information.

Conclusions:

• The proposed associative memory-enhanced model significantly improves X-ray report generation.
• Integrating global and local visual information with historical report data enhances clinical accuracy.
• The model offers a promising approach for more comprehensive and accurate AI-driven medical reporting.