Caution Regarding ChatGPT's Appropriateness and Reliability Regarding Surgery for Wrist Arthritis

Area of Science:

• Orthopedic Surgery
• Artificial Intelligence in Medicine
• Medical Information Quality

Background:

• Artificial intelligence (AI) tools like ChatGPT are increasingly used for information retrieval.
• The quality and consistency of AI-generated medical information require careful evaluation.
• This study focuses on AI's output concerning common wrist arthritis surgical procedures.

Purpose of the Study:

• To analyze the medical accuracy and consistency of ChatGPT-generated information on wrist arthritis procedures.
• To assess the readability and quality of AI-provided medical content.
• To compare AI performance across different surgical interventions for wrist arthritis.

Main Methods:

• 32 standardized questions on wrist osteoarthritis and procedures (4-corner fusion, proximal row carpectomy, etc.) were posed to ChatGPT-3.5.
• Responses were evaluated for medical accuracy by three independent reviewers.
• Quality and readability were assessed using DISCERN, Flesch-Kincade, and Flesch Reading Ease scores.

Main Results:

• 75% of ChatGPT responses were rated as appropriate, with excellent reliability (ICC=0.97).
• The AI provided good quality information (DISCERN=60) but at a college reading level (Flesch-Kincade Grade 14.6).
• Information accuracy varied, with proximal row carpectomy and total wrist arthroplasty being more reliable than 4-corner fusion and denervation.

Conclusions:

• ChatGPT demonstrates high reliability but variable accuracy for wrist arthritis procedures.
• The generic nature of some AI responses limits practical clinical usefulness.
• Cautious use of AI-generated medical information, with awareness of its limitations, is essential.