KeyCap3D: Keyword-Guided 3D Medical Image Captioning with Cross-Attention

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Radiology

Background:

• Automated radiological report generation is crucial for efficient clinical decision-making.
• Integrating multi-modal data, such as MRI images and clinical keywords, can improve report accuracy.

Purpose of the Study:

• To develop a keyword-guided cross-attention framework for automated radiological report generation from 3D FLAIR MRI brain tumor images.
• To enhance the fusion of visual MRI features with hierarchical clinical terminology for improved report generation.

Main Methods:

• Utilized M3D-CLIP as the image encoder and fine-tuned KeyBERT and BioBERT for hierarchical keyword extraction.
• Employed six cross-attention layers for multi-modal fusion across four hierarchical levels (abnormality type, lesion characteristics, anatomical location, lateralization).
• A four-layer transformer decoder autoregressively generated captions using enriched image-keyword representations.

Main Results:

• Achieved a significant reduction in training loss from 4.16 to 1.33.
• Demonstrated strong performance with BLEU-1 (0.5359), BLEU-2 (0.3969), and ROUGE-L (0.5051) scores on the BraTS2020 dataset.
• Generated captions accurately captured essential clinical information for decision support.

Conclusions:

• The proposed keyword-guided cross-attention framework effectively integrates visual and textual data for automated radiological report generation.
• This approach shows promise for improving the efficiency and accuracy of brain tumor diagnosis and reporting.
• The transformer-based generation conditioned on enriched representations offers a robust method for clinical decision support applications.