Multimodal Integrated Knowledge Transfer to Large Language Models through Preference Optimization with Biomedical...

Area of Science:

• Biomedical informatics
• Artificial intelligence
• Machine learning

Background:

• High-quality multimodal biomedical data is scarce, limiting the fine-tuning of Large Language Models (LLMs) for specialized tasks.
• Existing methods struggle to effectively transfer knowledge from multimodal data to unimodal LLMs.

Purpose of the Study:

• To introduce MINT (Multimodal Integrated Knowledge Transfer), a novel framework for aligning unimodal LLMs with multimodal biomedical data.
• To enhance the performance of LLMs in biomedical predictive tasks using preference optimization.

Main Methods:

• MINT aligns unimodal decoder LLMs with domain-specific patterns from multimodal data using preference optimization, primarily Odds Ratio Preference Optimization (ORPO).
• It leverages upstream multimodal machine learning (MML) models to transfer knowledge to downstream text-only or image-only LLMs.
• Demonstrated through rare genetic disease prediction (text input) and tissue type classification (image input).

Main Results:

• The MINT-derived model for rare genetic disease prediction outperformed SFT, RAG, DPO, and larger foundation models, using only text input.
• MINT significantly improved tissue type classification performance for an image-only LLM by leveraging a vision-language foundation model.
• The framework enables LLMs to perform predictive tasks with unimodal input while retaining multimodal knowledge.

Conclusions:

• MINT offers an effective strategy for aligning unimodal LLMs with multimodal expertise via preference optimization.
• The study highlights a hybrid approach combining encoder strengths with decoder models to improve biomedical LLM reasoning and reduce hallucinations.