Advancing conversational diagnostic AI with multimodal reasoning

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Multimodal Data Analysis

Background:

• Real-world clinical practice integrates diverse data types, including patient history, medical imagery, and documents.
• Current large language models (LLMs) for diagnostic dialogue are primarily text-based, limiting their applicability in complex, multimodal clinical scenarios.
• Evaluating diagnostic AI requires assessing its ability to handle and reason with multimodal information.

Purpose of the Study:

• To introduce a multimodal extension of the Articulate Medical Intelligence Explorer (multimodal AMIE) capable of processing and reasoning with diverse data types.
• To develop a state-aware dialogue framework for dynamic history-taking based on diagnostic uncertainty.
• To evaluate the diagnostic accuracy and conversational quality of multimodal AMIE against primary care physicians in simulated telehealth settings.

Main Methods:

• Development of a state-aware dialogue framework enabling multimodal data interpretation and reasoning.
• A randomized, blinded exploratory study involving 105 simulated telehealth consultations.
• Inclusion of diverse data modalities: dermatology photographs, electrocardiograms, and clinical documents.
• Evaluation by 18 specialist physicians assessing diagnostic accuracy and conversation quality.

Main Results:

• Multimodal AMIE outperformed primary care physicians (PCPs) in diagnostic accuracy and conversation quality.
• The AI system demonstrated superior performance across 29 of 32 evaluation axes.
• Specifically, multimodal AMIE showed enhanced capabilities in seven of nine metrics assessing multimodal reasoning.
• The AI exhibited higher scores in history-taking and empathy compared to PCPs.

Conclusions:

• State-aware reasoning effectively integrates text and visual information for improved diagnostic capabilities.
• Multimodal AI systems show significant potential to augment clinicians in complex, multimodal diagnostic environments.
• The findings support the development of AI that can handle the multifaceted nature of modern healthcare delivery.