Appropriateness and Consistency of an Online Artificial Intelligence System's Response to Common Questions Regarding Cervical Fusion

Mark Miller ¹, William T DiCiurcio ¹, Matthew Meade ¹

⁰Jefferson Health-New Jersey, Stratford.

Clinical Spine Surgery +

|

February 10, 2025

View abstract on PubMed

Summary

ChatGPT provides moderately accurate responses to cervical surgery questions, but reliability is poor. Further AI research is needed to improve the quality and consistency of these AI-generated answers for patient inquiries.

Area Of Science

Artificial Intelligence in Medicine
Natural Language Processing in Healthcare
Spine Surgery Knowledge Dissemination

Background

Artificial Intelligence (AI) and machine learning are transforming scientific research and clinical practice.
Large language models like ChatGPT show promise in answering medical questions, with prior studies validating responses for orthopedic procedures.
A knowledge gap exists regarding ChatGPT's accuracy and reliability for cervical surgery-related inquiries.

Purpose Of The Study

To evaluate the accuracy and utility of ChatGPT responses to common cervical surgery questions.
To identify the quality and consistency of information provided by an AI language model on this specific surgical domain.

Main Methods

A prospective survey study design was employed.
Twenty distinct cervical surgery questions were posed to ChatGPT-3.5 three times each (totaling 60 responses).
Responses were assessed for accuracy and utility by three fellowship-trained spine surgeons using a 5-point rating scale, with intraclass correlation coefficients (ICCs) calculated for reliability.

Main Results

The average quality score for ChatGPT responses was 3.17, with 66.7% of responses rated as at least "moderate" quality by at least one reviewer.
Forty-five percent of questions received "moderate" or higher quality ratings from all three reviewers.
Test-retest reliability was poor, indicated by an ICC of 0.0941.

Conclusions

ChatGPT can generate moderately accurate responses to common patient questions about cervical surgery.
The current iteration of ChatGPT exhibits poor consistency in its responses regarding cervical surgery.
Further research and development in AI are essential to enhance the reliability and accuracy of AI-generated medical information.