Evaluating Chat Generative Pre-trained Transformer Responses to Common Pediatric In-toeing Questions

Area of Science:

• Pediatric Orthopaedics
• Health Informatics
• Artificial Intelligence in Healthcare

Background:

• Patients increasingly use AI for health information, raising concerns about accuracy.
• In-toeing is a common pediatric condition often managed conservatively.
• Evaluating AI's role in patient education for conditions like in-toeing is crucial.

Purpose of the Study:

• To assess the accuracy and reproducibility of ChatGPT's responses regarding pediatric in-toeing.
• To evaluate the quality of information provided by ChatGPT for a common pediatric orthopaedic condition.
• To contribute to understanding AI's utility and limitations in patient education.

Main Methods:

• Compiled 25 common pediatric in-toeing questions from healthcare websites.
• Queried ChatGPT 3.5 and assessed responses for accuracy and clarity.
• Two pediatric orthopaedic surgeons evaluated responses; reproducibility was tested.

Main Results:

• 46% of ChatGPT responses were "excellent," and 44% were "satisfactory (minimal clarification)."
• Responses were consistently accurate regarding in-toeing being normal/resolving (62%) and recommending provider evaluation (100%).
• Readability was appropriate (Flesch-Kincaid 4.9), but response complexity was collegiate (12.7); moderate reproducibility was observed (Spearman rho=0.55).

Conclusions:

• ChatGPT offers a serviceable, though imperfect, representation of pediatric in-toeing diagnosis and management.
• The AI demonstrated moderate response consistency.
• Improving ChatGPT's readability and incorporating evidence-based guidelines could enhance its utility for patient education.