Translating medical image to radiological report: Adaptive multilevel multi-attention approach

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Medical imaging is crucial for disease diagnosis.
• Radiologists can benefit from automated report generation systems.
• Existing methods lack domain-specific knowledge, impacting report credibility.

Purpose of the Study:

• To develop an Adaptive Multilevel Multi-Attention (AMLMA) approach for generating radiological reports from chest X-ray images.
• To improve the accuracy and credibility of automated medical report generation.
• To incorporate domain-specific visual-textual knowledge into the translation system.

Main Methods:

• Utilized an encoder-decoder framework with adaptive attention mechanisms.
• Employed a Convolutional Neural Network (CNN) with Residual Attention Module (RAM) for visual encoding.
• Integrated Word2Vec and FastText embeddings with a Bi-directional Long Short-Term Memory (Bi-LSTM) network for textual encoding.
• Developed weighted multilevel associations of visual-semantic and visual-based linguistic attention for decoding.

Main Results:

• The CNN with RAM improved multi-label abnormality detection (Recall: 0.4423, Precision: 0.1803, F1-score: 0.2551).
• The AMLMA model outperformed state-of-the-art approaches in language generation metrics (Bleu-4: 0.172, Meteor: 0.247, Rouge_L: 0.376, CIDEr: 0.381).
• Introduced a 'Unique Index' (UI) statistic to measure the generation of unique reports.

Conclusions:

• The AMLMA architecture effectively understands X-ray views and generates relevant radiography statements.
• The model balances visual and linguistic information using multi-level knowledge and adaptive attention.
• The proposed approach generates admissible and informative radiology reports.