Enhancing diagnostic capability with multi-agents conversational large language models

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing

Background:

• Large Language Models (LLMs) show potential in healthcare but struggle with complex medical reasoning.
• Current LLM applications often lack the collaborative and iterative nature of clinical practice.

Purpose of the Study:

• To develop and evaluate a Multi-Agent Conversation (MAC) framework for improving disease diagnosis using LLMs.
• To compare the diagnostic performance of the MAC framework against single LLM models and other advanced AI methods.

Main Methods:

• A MAC framework was designed, simulating Multi-Disciplinary Team discussions with specialized AI agents.
• The framework was evaluated on 302 rare disease cases using GPT-3.5 and GPT-4 as base models.
• Performance was compared against baseline LLMs, Chain of Thought (CoT), Self-Refine, and Self-Consistency methods.

Main Results:

• The MAC framework demonstrated superior diagnostic accuracy and test suggestion relevance compared to single LLMs in both initial and follow-up consultations.
• Optimal performance was achieved with a configuration of four doctor agents and one supervisor agent, utilizing GPT-4.
• MAC consistently outperformed other comparative methods, including CoT, Self-Refine, and Self-Consistency, in accuracy and output volume.

Conclusions:

• The Multi-Agent Conversation framework significantly enhances the diagnostic capabilities of LLMs in healthcare.
• This approach effectively bridges the gap between theoretical medical knowledge and practical clinical application.
• Multi-agent LLM systems hold substantial promise for advancing clinical decision support and warrant further research for implementation.