Appropriateness and Consistency of an Online Artificial Intelligence System's Response to Common Questions Regarding Cervical Fusion
View abstract on PubMed
Summary
This summary is machine-generated.ChatGPT provides moderately accurate responses to cervical surgery questions, but reliability is poor. Further AI research is needed to improve the quality and consistency of these AI-generated answers for patient inquiries.
Area Of Science
- Artificial Intelligence in Medicine
- Natural Language Processing in Healthcare
- Spine Surgery Knowledge Dissemination
Background
- Artificial Intelligence (AI) and machine learning are transforming scientific research and clinical practice.
- Large language models like ChatGPT show promise in answering medical questions, with prior studies validating responses for orthopedic procedures.
- A knowledge gap exists regarding ChatGPT's accuracy and reliability for cervical surgery-related inquiries.
Purpose Of The Study
- To evaluate the accuracy and utility of ChatGPT responses to common cervical surgery questions.
- To identify the quality and consistency of information provided by an AI language model on this specific surgical domain.
Main Methods
- A prospective survey study design was employed.
- Twenty distinct cervical surgery questions were posed to ChatGPT-3.5 three times each (totaling 60 responses).
- Responses were assessed for accuracy and utility by three fellowship-trained spine surgeons using a 5-point rating scale, with intraclass correlation coefficients (ICCs) calculated for reliability.
Main Results
- The average quality score for ChatGPT responses was 3.17, with 66.7% of responses rated as at least "moderate" quality by at least one reviewer.
- Forty-five percent of questions received "moderate" or higher quality ratings from all three reviewers.
- Test-retest reliability was poor, indicated by an ICC of 0.0941.
Conclusions
- ChatGPT can generate moderately accurate responses to common patient questions about cervical surgery.
- The current iteration of ChatGPT exhibits poor consistency in its responses regarding cervical surgery.
- Further research and development in AI are essential to enhance the reliability and accuracy of AI-generated medical information.

