Mapping medical image-text to a joint space via masked modeling

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Computer Vision
• Natural Language Processing

Background:

• Masked autoencoders excel at feature extraction in vision and language domains.
• Existing methods lack effective joint representation learning for medical images and text.

Purpose of the Study:

• To introduce a novel self-supervised learning paradigm, multi-modal masked autoencoders (M³AE), for medical vision-and-language representation learning.
• To adapt masked autoencoder techniques for the unique challenges of medical data.

Main Methods:

• M³AE reconstructs pixels and tokens from masked medical images and texts into a joint embedding space.
• Employs differential masking ratios for images and text, leveraging multi-layer features.
• Utilizes distinct vision and language decoders tailored for medical data.

Main Results:

• M³AE achieves state-of-the-art performance across all evaluated downstream medical vision-and-language tasks.
• Experimental results validate the effectiveness of M³AE's distinct architectural components.
• The approach demonstrates robust feature extraction capabilities for multimodal medical data.

Conclusions:

• M³AE offers a powerful self-supervised approach for medical vision-and-language representation learning.
• The method successfully integrates image and text data, improving downstream task performance.
• Further research can explore M³AE's potential in diverse clinical applications.