Large Language Model Influence on Management Reasoning: A Randomized Controlled Trial.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Medical Education and Training

Background:

• Large language models (LLMs) show potential in diagnostic reasoning, but their impact on complex management decisions is not well understood.
• Physician performance in open-ended clinical management tasks often relies on synthesizing information from various sources.

Purpose of the Study:

• To evaluate whether LLM assistance enhances physician performance in open-ended clinical management reasoning tasks.
• To compare the effectiveness of LLM-augmented resources versus conventional resources alone.

Main Methods:

• A prospective, randomized controlled trial involving 92 physicians (attendings and residents) across multiple institutions.
• Participants used either GPT-4 (via ChatGPT Plus) with conventional resources or conventional resources alone to manage five expert-developed clinical case vignettes.
• Scoring rubrics were created via a Delphi process to assess management and diagnostic decisions.

Main Results:

• Physicians using LLM assistance achieved significantly higher overall scores (6.5% increase, p<0.001).
• Improvements were noted in management (6.1%), diagnostic (12.1%), and case-specific (6.2%) decision-making domains.
• LLM users spent more time per case (119.3 seconds), with no significant difference between GPT-4-augmented and GPT-4 alone groups.

Conclusions:

• LLM assistance demonstrably improves physician management reasoning, especially in contextual and patient-specific decision-making.
• These findings suggest LLMs can serve as valuable tools to augment clinical management in complex medical scenarios.
• The integration of AI tools like LLMs holds promise for enhancing physician performance and patient care.