Auxiliary signal-guided knowledge encoder-decoder for medical report generation.

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Natural Language Processing

Background:

• Medical report generation is crucial for clinical decision-making, especially during pandemics.
• Existing methods struggle with fine-grained detail, semantic coherence, and data bias in medical image descriptions.
• Current approaches often overlook the localized regions of interest for radiologists, treating the entire image as relevant.

Purpose of the Study:

• To develop an advanced model for generating accurate and semantically coherent medical reports from images.
• To address limitations in current medical report generation models, including data bias and focus on irrelevant image regions.
• To mimic the working patterns of radiologists for more effective medical image analysis and reporting.

Main Methods:

• Proposed an Auxiliary Signal-Guided Knowledge Encoder-Decoder (ASGK) model.
• Integrated auxiliary patches to expand visual features for the Transformer encoder.
• Utilized external linguistic signals to enhance the decoder's prior knowledge during pre-training.

Main Results:

• The ASGK model demonstrated strong performance on multiple medical report generation benchmarks (CX-CHR, IU X-Ray, COV-CTR).
• The approach significantly improved medical report generation compared to previous methods.
• Transformer-based models incorporating auxiliary signals showed superior capabilities in medical terminology classification and paragraph generation.

Conclusions:

• Combining auxiliary signals with Transformer architecture significantly enhances medical report generation.
• The ASGK model effectively addresses data bias and improves the focus on diagnostically relevant image regions.
• Auxiliary signal-driven Transformer models offer a promising advancement for automated medical reporting.