Evaluating Artificial Intelligence-Generated Responses to Patient Questions Regarding Orthobiologic Injections

Area of Science:

• Orthopaedics
• Biotechnology
• Artificial Intelligence

Background:

• Patient interest in orthobiologic injections is increasing.
• Patients increasingly use AI large language models (LLMs) for health information.
• The accuracy and readability of AI-generated orthobiologic information for patients are unclear.

Purpose of the Study:

• To assess the accuracy and readability of responses from 3 popular AI LLMs (ChatGPT, Gemini, Grok) to common patient questions about orthobiologic injections.

Main Methods:

• Cross-sectional study evaluating 20 common patient questions across ChatGPT 4o, Gemini 2.5 Flash, and Grok 3.
• Four independent reviewers assessed response accuracy using the ChatGPT Response Rating System (CRRS) and AI Response Metric (AIRM).
• Readability was measured using the Flesch-Kincaid Grade Level (FKGL).

Main Results:

• Response accuracy was generally acceptable, but 25-50% required clarification.
• Gemini demonstrated superior accuracy compared to ChatGPT (CRRS, P=.04; AIRM, P=.03).
• All AI models generated responses at a collegiate reading level or higher, exceeding recommended patient education levels.

Conclusions:

• AI LLMs provide generally accurate orthobiologic information but lack patient-appropriate readability.
• Gemini showed improved accuracy over ChatGPT, yet all models exhibited clarity limitations.
• AI responses should supplement, not replace, physician-led patient education.