A Proactive Agent Collaborative Framework for Zero-Shot Multimodal Medical Reasoning.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Informatics
• Computational Health

Background:

• Large language models (LLMs) show promise in healthcare but lack domain-specific knowledge and multimodal capabilities.
• Current LLMs are limited by text-only inputs and insufficient medical reasoning skills.

Purpose of the Study:

• To introduce MultiMedRes, a novel multimodal medical collaborative reasoning framework.
• To enhance LLM performance in healthcare by simulating physician communication and knowledge acquisition.

Main Methods:

• MultiMedRes employs a learner agent that decomposes problems, interacts with expert models for domain-specific knowledge, and integrates information.
• The framework utilizes an "Inquire, Interact, Integrate" process for multimodal reasoning.
• Validation was performed on X-ray image-based visual question answering tasks.

Main Results:

• MultiMedRes achieved state-of-the-art zero-shot performance on difference visual question answering for X-ray images.
• The framework surpassed the performance of fully supervised methods.
• Demonstrated the potential for trustworthy and interpretable AI assistance in clinical settings.

Conclusions:

• MultiMedRes effectively addresses the limitations of unimodal LLMs in medical reasoning.
• The framework facilitates human-AI collaboration for tasks like monitoring patient treatment progression.
• Paves the way for advanced AI tools in clinical decision support.