FreeMMIF: interactive multimodal medical image fusion via instruction-aware diffusion

Area of Science:

• Medical imaging
• Artificial intelligence in medicine
• Computer vision

Background:

• Multimodal medical image fusion (MMIF) integrates functional and metabolic data for improved clinical diagnosis.
• Current MMIF methods lack flexibility and on-the-fly adjustment capabilities, hindering clinical application.
• Existing techniques often require reference images for supervision, which are not always available.

Purpose of the Study:

• To develop an interactive framework (FreeMMIF) for preference-adaptive MMIF.
• To enable physicians to control fusion parameters through natural language instructions.
• To provide robust supervision for MMIF models without relying on reference images.

Main Methods:

• Utilized a vision-language model (VLM) for pseudo-ground truth generation and instruction-based weight control.
• Employed an instruction-aware diffusion model with adaptive feature re-weighting for dynamic fusion.
• Implemented prompt engineering for a VLM to translate physician instructions into fusion parameter adjustments.

Main Results:

• FreeMMIF generates diagnostic-quality fused images that align with clinical intent.
• The framework successfully integrates VLM-based pseudo-labeling and instruction-aware diffusion.
• Physician control over source modality ratios was achieved through text instructions.

Conclusions:

• FreeMMIF offers a flexible and interactive solution for multimodal medical image fusion.
• The proposed approach enhances diagnostic reliability by adapting fusion to specific clinical needs.
• This work advances the integration of AI, particularly VLMs and diffusion models, into clinical image analysis.