Spatio-Temporal and Retrieval-Augmented Modeling for Chest X-Ray Report Generation.

Area of Science:

• Medical Imaging Analysis
• Artificial Intelligence in Healthcare
• Natural Language Generation

Background:

• Automatic chest X-ray report generation is a growing research area.
• Existing methods often overlook temporal dynamics and fixed image inputs.
• Clinical diagnosis integrates historical and current patient data for accurate interpretation.

Purpose of the Study:

• To propose STREAM (Spatio-Temporal and REtrieval-Augmented Modelling) for automatic chest X-ray report generation.
• To incorporate temporal information from historical studies and spatial information from multi-view images.
• To mimic clinical diagnosis by integrating current and historical imaging data.

Main Methods:

• Employs an encoder-decoder architecture with a large language model (LLM) as the decoder.
• A token packer captures spatio-temporal visual dynamics for flexible image fusion.
• A progressive semantic retriever augments generation with knowledge bank entities and regional details.

Main Results:

• The proposed method achieves state-of-the-art performance on public datasets.
• STREAM effectively integrates spatio-temporal visual dynamics and retrieved textual prompts.
• The knowledge bank encapsulates structured anatomical chest X-ray knowledge.

Conclusions:

• STREAM demonstrates superior performance in automatic chest X-ray report generation.
• The integration of temporal dynamics and retrieval-augmented knowledge significantly enhances report quality.
• The method offers a promising approach for clinical decision support.