Retrieval-augmented ChatGPT-4o improves accuracy but reduces readability in hip arthroscopy patient education

Area of Science:

• Orthopaedic Surgery
• Artificial Intelligence in Medicine
• Health Informatics

Background:

• Large language models (LLMs) show promise for patient education but their reliability in specialized medical fields like orthopaedics is uncertain.
• Evaluating different LLM configurations is crucial for understanding their potential and limitations in providing accurate patient information.

Purpose of the Study:

• To compare the accuracy, readability, and patient-centeredness of responses from standard ChatGPT-4o and its 'deep research' mode.
• To assess the suitability of these AI models for hip arthroscopy patient education.

Main Methods:

• Thirty standardized patient questions on hip arthroscopy were used.
• Responses from standard ChatGPT-4o and its 'deep research' mode were evaluated by two orthopaedic surgeons.
• Assessments included accuracy, clarity, comprehensiveness, and readability using Likert scales and Flesch-Kincaid metrics.

Main Results:

• 'Deep research' mode demonstrated significantly higher accuracy (p=0.012) and comprehensiveness (p<0.001).
• Standard ChatGPT-4o showed superior clarity (p=0.048) and readability scores (Flesch-Kincaid Grade Level and Reading Ease Score; p<0.001).
• Readability Likert scores were comparable between models (p=0.729).

Conclusions:

• 'Deep research' mode offers greater scientific rigor, while the standard model excels in readability for hip arthroscopy patient education.
• A hybrid approach may enhance educational effectiveness, but clinical oversight is essential to prevent misinformation.
• Findings suggest modest differences and highlight the accuracy-readability trade-off in LLMs, warranting further exploratory research.