Appropriateness and Consistency of an Online Artificial Intelligence System's Response to Common Questions Regarding...

Prospective survey study.

To address a gap that exists concerning ChatGPT's ability to respond to various types of questions regarding cervical surgery.

Artificial Intelligence (AI) and machine learning have been creating great change in the landscape of scientific research. Chat Generative Pre-trained Transformer(ChatGPT), an online AI language model, has emerged as a powerful tool in clinical medicine and surgery. Previous studies have demonstrated appropriate and reliable responses from ChatGPT concerning patient questions regarding total joint arthroplasty, distal radius fractures, and lumbar laminectomy. However, there is a gap that exists in examining how accurate and reliable ChatGPT responses are to common questions related to cervical surgery.

Twenty questions regarding cervical surgery were presented to the online ChatGPT-3.5 web application 3 separate times, creating 60 responses. Responses were then analyzed by 3 fellowship-trained spine surgeons across 2 institutions using a modified Global Quality Scale (1-5 rating) to evaluate accuracy and utility. Descriptive statistics were reported based on responses, and intraclass correlation coefficients were then calculated to assess the consistency of response quality.

Out of all questions proposed to the AI platform, the average score was 3.17 (95% CI, 2.92, 3.42), with 66.7% of responses being recorded to be of at least "moderate" quality by 1 reviewer. Nine (45%) questions yielded responses that were graded at least "moderate" quality by all 3 reviewers. The test-retest reliability was poor with the intraclass correlation coefficient (ICC) calculated as 0.0941 (-0.222, 0.135).

This study demonstrated that ChatGPT can answer common patient questions concerning cervical surgery with moderate quality during the majority of responses. Further research within AI is necessary to increase response.