Towards conversational artificial intelligence for disease management

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing

Background:

• Large language models (LLMs) show potential in diagnostic dialogue but require further exploration for clinical management reasoning.
• Effective clinical management involves complex reasoning about disease progression, treatment response, and medication safety.

Purpose of the Study:

• To advance the diagnostic capabilities of the Articulate Medical Intelligence Explorer (AMIE) for multi-visit clinical management and dialogue.
• To evaluate AMIE's performance against primary care physicians (PCPs) in clinical management reasoning and medication prescription.

Main Methods:

• Developed a new LLM-based agentic system (AMIE) leveraging long-context capabilities for clinical knowledge grounding.
• Conducted a randomized, blinded virtual Objective Structured Clinical Examination (OSCE) comparing AMIE to 21 PCPs across 100 multi-visit cases.
• Utilized the RxQA benchmark, derived from national drug formularies, to assess medication reasoning.

Main Results:

• AMIE demonstrated non-inferiority to PCPs in management reasoning, as assessed by specialists.
• AMIE scored higher than PCPs in the preciseness of treatments and investigations, and guideline adherence.
• AMIE outperformed PCPs on higher difficulty medication reasoning questions, especially when accessing external drug information.

Conclusions:

• AMIE represents a significant advancement in conversational AI for disease management, showing strong performance in clinical reasoning and medication safety.
• The system's ability to ground reasoning in clinical guidelines and drug formularies enhances its reliability.
• While further research is needed for real-world application, AMIE shows potential as a valuable tool for healthcare professionals.