Evaluation of ChatGPT-4o's answers to questions about hip arthroscopy from the patient perspective

Gökhan Ayık ¹, Niyazi Ercan , Yunus Demirtaş

⁰Yüksek İhtisas Üniversitesi Ortopedi ve Travmatoloji Anabilim Dalı, 06530 Çankaya, Ankara, Türkiye. drgayik@gmail.com.

Joint Diseases and Related Surgery +

|

December 25, 2024

View abstract on PubMed

Summary

ChatGPT-4o provides high-quality answers to common hip arthroscopy questions, scoring above average in relevance, accuracy, and clarity. While helpful, patient consultation with orthopedic specialists remains essential for final treatment decisions.

Area Of Science

Orthopedic Surgery
Artificial Intelligence in Medicine
Patient Education

Background

Hip arthroscopy is a common orthopedic procedure.
Patients frequently seek information online about hip arthroscopy.
The quality of AI-generated responses for medical queries is an emerging area of research.

Purpose Of The Study

To evaluate the quality of responses generated by ChatGPT-4o for frequently asked patient questions about hip arthroscopy.
To assess the relevance, accuracy, clarity, and completeness of AI-generated medical information.

Main Methods

Identified the top 20 patient questions on hip arthroscopy using Google search data.
Submitted these questions to ChatGPT-4o for response generation.
Ten orthopedic surgeons rated the AI responses on a 1-5 scale for quality metrics.
Assessed interrater reliability using the intraclass correlation coefficient (ICC).

Main Results

ChatGPT-4o responses received high average scores (4.49/5) across all evaluated metrics.
Accuracy and clarity scored highest, while completeness received the lowest scores.
Interrater reliability for surgeon agreement was generally insufficient (ICC=0.004).

Conclusions

ChatGPT-4o demonstrates the capacity to provide above-average quality information on hip arthroscopy for patient inquiries.
Despite high AI response quality, professional consultation with orthopedic specialists is crucial for patient care.
AI tools can supplement, but not replace, expert medical advice in orthopedic decision-making.