ChatGPT-4 Knows Its A B C D E but Cannot Cite Its Source

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Trauma Surgery

Background:

• Artificial intelligence language models like ChatGPT show promise as accessible educational tools in orthopaedic surgery.
• However, the accuracy of references provided by AI models is a critical concern for medical content integrity.
• This study focuses on the reliability of ChatGPT-4's references for the ABCDE trauma protocol.

Purpose of the Study:

• To evaluate the accuracy of references generated by ChatGPT-4 for the Airway, Breathing, Circulation, Disability, Exposure (ABCDE) approach in trauma surgery.
• To assess the reliability and reproducibility of AI-generated medical references.

Main Methods:

• Two independent reviewers assessed 30 ChatGPT-4 generated references for the ABCDE trauma protocol.
• References were graded as nonexistent, inaccurate, or accurate.
• Cohen's Kappa coefficient and descriptive statistics were used to analyze inter-rater agreement and mean accuracy.

Main Results:

• ChatGPT-4 achieved an average reference accuracy score of 66.7%.
• Only 43.3% of the 30 references were accurate; 56.7% were inaccurate or nonexistent.
• Reference accuracy was consistent across the five trauma protocol categories (p = 0.437).

Conclusions:

• ChatGPT-4's high rate of inaccurate or nonexistent references (57%) raises safety concerns for professional medical decision-making without verification.
• The AI model can serve as an adjunct learning tool if used cautiously with cross-referencing to enhance knowledge rehearsal.