A randomized controlled trial on evaluating clinician-supervised generative AI for decision support

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Emergency Medicine

Background:

• Generative AI integration into telemedicine as clinical decision support systems (CDSS) offers potential for improved outcomes but is under-researched.
• Current applications of AI in clinical decision-making, particularly in emergency scenarios, require further investigation.

Purpose of the Study:

• To evaluate the efficacy of ChatGPT, a generative AI tool, in providing clinical guidance during cardiac arrest simulations.
• To compare the performance, cognitive load, and trust associated with traditional paper guides, autonomous ChatGPT, and clinician-supervised ChatGPT.

Main Methods:

• Fifty-four participants without medical backgrounds engaged in randomized controlled trials using an Augmented Reality (AR) headset for a CPR scenario.
• Intervention groups included a paper guide, autonomous ChatGPT, and clinician-supervised ChatGPT.
• Performance, physiological metrics (LF/HF ratio), and self-reported trust were recorded.

Main Results:

• The clinician-supervised ChatGPT group demonstrated significantly higher decision accuracy than the paper guide and autonomous ChatGPT groups.
• Physiological data indicated a potentially lower cognitive load in the supervised group, evidenced by a reduced LF/HF ratio.
• Trust in AI was highest in the supervised condition, though response time was longer and autonomous AI suggested a risky option.

Conclusions:

• Supervised generative AI shows promise for enhancing decision accuracy and user trust in emergency healthcare.
• Clinician oversight is crucial for safe and effective AI implementation in critical care settings.
• Further research is needed to optimize AI supervision strategies and evaluate real-world clinical implementation.