Medical Vision-Language Pre-Training for Brain Abnormalities

Area of Science:

• Multimodal AI
• Medical Imaging Analysis
• Natural Language Processing

Background:

• Vision-language models (VLMs) are powerful but often lack domain-specific medical knowledge.
• Existing VLMs struggle with the nuances of clinical data, limiting their application in healthcare.
• There is a need for specialized VLMs trained on medical data for accurate clinical AI tasks.

Purpose of the Study:

• To demonstrate a method for automatically collecting medical image-text aligned data for pre-training VLMs.
• To develop a high-performance VLM tailored for specific medical tasks, using brain abnormalities as a case study.
• To address the challenge of mapping subfigures to subcaptions in medical imaging reports.

Main Methods:

• Developed a pipeline to collect a large brain image-text dataset from public resources like PubMed, case reports, and journals.
• Constructed a specialized vision-language model using the curated dataset for pre-training.
• Investigated and addressed the challenge of aligning image subfigures with their corresponding subcaptions.

Main Results:

• Successfully created a pipeline for automated collection of medical image-text data.
• Developed a high-performance vision-language model pre-trained on brain abnormality data.
• The resulting dataset and code are publicly available for further research.

Conclusions:

• The proposed pipeline effectively streamlines the creation of domain-specific VLMs for medical applications.
• This approach enables the development of more accurate and reliable clinical AI tools.
• The publicly available resources facilitate advancements in multimodal clinical AI research.