Utilizing Artificial Intelligence and Chat Generative Pretrained Transformer to Answer Questions About Clinical...

Area of Science:

• Neuroscience
• Artificial Intelligence
• Clinical Guidelines

Background:

• Clinical guidelines are essential for evidence-based medical practice.
• Artificial intelligence (AI) chatbots like ChatGPT are increasingly used for information retrieval.
• The reliability of AI in providing accurate clinical information is a growing concern.

Purpose of the Study:

• To evaluate the accuracy of ChatGPT in responding to questions based on Society for Neuroscience in Anesthesiology and Critical Care (SNACC) clinical guidelines.
• To assess ChatGPT's ability to incorporate high-quality recommendations (HQRs) from SNACC guidelines on stroke and spine surgery.
• To identify potential inaccuracies or harmful recommendations provided by ChatGPT.

Main Methods:

• Four neuroanesthesiologists independently reviewed ChatGPT responses to 52 HQRs from three SNACC guidelines.
• Assessments included the presence of HQRs, incorrect references, and adherence to guidelines.
• ChatGPT's responses to generic and targeted questions were analyzed.

Main Results:

• Reviewer agreement on HQRs varied widely (0%–100%).
• Only 8% of HQRs were consistently identified after generic questions; this rose to 44% after targeted questions.
• Potentially harmful recommendations were found, and ChatGPT failed to cite the SNACC guidelines.

Conclusions:

• ChatGPT responses require human interpretation and are not consistently accurate regarding HQRs.
• Even with targeted questioning, less than 50% of HQRs were reliably identified.
• Current AI chatbots like ChatGPT are not recommended for clinical decision-making due to reliability concerns.