Assessing information provided via artificial intelligence regarding distal biceps tendon repair surgery

Area of Science:

• Orthopedic Surgery
• Artificial Intelligence in Medicine
• Medical Information Quality

Background:

• Patients increasingly use online resources for medical information.
• Artificial Intelligence (AI) chatbots like ChatGPT are emerging as potential information sources.
• Evaluating the reliability and readability of AI-generated medical content is crucial.

Purpose of the Study:

• To analyze the quality, accuracy, reliability, and readability of ChatGPT's information on distal biceps tendon repair surgery.
• To assess AI-generated patient information against established medical benchmarks.

Main Methods:

• ChatGPT 3.5 answered 27 patient questions on distal biceps repair surgery.
• Answers were categorized by Rothwell criteria (Fact, Policy, Value).
• Evaluated using DISCERN, JAMA benchmark criteria, Flesch-Kincaid Reading Ease Score (FRES), and Flesch-Kincaid Grade Level (FKGL).

Main Results:

• DISCERN scores indicated 'good' quality (59-61) across categories.
• JAMA benchmark criteria scores were 0 (lowest) for all categories.
• FRES scores indicated difficult readability (21.77-24.49), with FKGL scores around 15 (college graduate level).

Conclusions:

• ChatGPT offers a 'good' quality source for distal biceps repair information compared to uncited online resources.
• However, accuracy and reliability were low, and readability was high (college-graduate level).
• Physicians should advise patients about ChatGPT's limitations for distal biceps repair information.