Biomedical Visual Instruction Tuning with Clinician Preference Alignment.

Area of Science:

• Artificial Intelligence
• Biomedical Informatics
• Machine Learning

Background:

• Multimodal foundation models show promise in visual and textual understanding.
• Adapting these models to specialized fields like biomedicine requires large, domain-specific instruction datasets.
• Existing automatic dataset curation methods lack explicit alignment with domain expertise.

Purpose of the Study:

• To propose a data-centric framework, BioMed-VITAL, for tuning biomedical multimodal foundation models.
• To incorporate clinician preferences in both generating and selecting instruction data.
• To enhance the performance of models in specialized biomedical applications.

Main Methods:

• Developed BioMed-VITAL, a framework integrating clinician preferences into data generation and selection.
• Utilized GPT-4V with clinician-selected demonstrations for generating preference-aligned data candidates.
• Trained a selection model to distill clinician and policy-guided preferences for high-quality data selection.
• Tuned biomedical multimodal foundation models using the curated instruction-following data.

Main Results:

• The model tuned with BioMed-VITAL data showed significant improvements in open visual chat (18.5% relative increase).
• Achieved a high win rate of up to 81.73% in medical Visual Question Answering (VQA).
• Demonstrated the effectiveness of clinician preference alignment in medical instruction tuning.

Conclusions:

• BioMed-VITAL effectively enhances biomedical multimodal foundation models by incorporating clinician preferences.
• The proposed data-centric approach leads to superior performance in specialized medical AI tasks.
• The developed dataset and models are publicly available to advance biomedical AI research.