Contrastive pre-training and linear interaction attention-based transformer for universal medical reports generation

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Automated medical report generation aids diagnosis and treatment by reducing workload and variability.
• Current image-encoder/language-decoder models face challenges in fusing visual and language cues and require domain-specific feature extractors.

Purpose of the Study:

• To develop an improved method for medical image report generation.
• To enhance the fusion of visual and language information in automated report generation.
• To address the domain gap for pre-trained image feature extractors in medical tasks.

Main Methods:

• Proposed a weighted query-key interacting attention module with first-order and second-order interactions for robust fusion.
• Introduced a contrastive pre-training step to minimize the domain gap between image encoders and target datasets.
• Validated the model on established datasets (IU X-RAY, MIMIC-CXR) and new retina datasets (Retina ImBank, Retina Chinese).

Main Results:

• The proposed weighted query-key attention module demonstrated superior fusion of language and visual signals compared to scaled dot-product attention.
• Contrastive pre-training effectively reduced the domain gap for image feature extraction.
• The model achieved state-of-the-art performance across all tested medical report generation datasets.

Conclusions:

• The developed method significantly enhances automated medical report generation by improving data fusion and addressing domain specificity.
• The new datasets (Retina ImBank, Retina Chinese) will serve as valuable benchmarks for future research in multi-modal medical report generation.