Token-Mixer: Bind Image and Text in One Embedding Space for Medical Image Reporting

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Automatic medical image reporting is a growing research area.
• Cross-modal alignment between medical images and diagnostic reports is crucial.
• Exposure bias in autoregressive text generation hinders model optimization.

Purpose of the Study:

• To propose a novel framework, Token-Mixer, for enhanced medical image reporting.
• To address the exposure bias problem in generating diagnostic reports.
• To improve cross-modal alignment between medical images and text.

Main Methods:

• Developed the Token-Mixer framework with an image encoder, text encoder, and text decoder.
• Encoded images and reports into tokens, then randomly mixed them.
• Utilized a tailored text decoder and alternative training strategy for optimization.
• Trained the model using image-text generation and text-text generation to mitigate exposure bias.

Main Results:

• Token-Mixer demonstrated enhanced image-text alignment.
• The framework achieved state-of-the-art performance on three public datasets.
• Experiments confirmed the effectiveness of the proposed approach.

Conclusions:

• Token-Mixer successfully binds image and text in a shared embedding space.
• The framework effectively enhances cross-modal alignment for medical image reporting.
• This approach offers a promising solution for automated diagnostic report generation.