Four large language models in parent-oriented education for pediatric flexible flatfoot: Quality, readability, hallucination, and reproducibility
Related Experiment Videos
Erkan Servet1, Beytullah Unat2, Musa Alperen Bilgin3
1Department of Orthopaedics and Traumatology, Faculty of Medicine, Gaziantep University, Gaziantep, Türkiye.
Purpose:
Parents increasingly consult large language model (LLM) chatbots about pediatric flexible flatfoot, a common condition that rarely requires treatment and in which the clinical task is chiefly communication. We compared four LLMs on the quality, readability, and reproducibility of parent-oriented answers.
Methods:
This was a cross-sectional, comparative content analysis of AI-generated text. No patients, clinical records, or human participants were involved; the units of analysis were parent-oriented questions and the chatbot responses they generated. Twenty parent-oriented questions were submitted in triplicate to ChatGPT, Claude, DeepSeek, and Gemini in independent, memory-disabled sessions (240 responses). Two blinded pediatric orthopedic surgeons scored responses (Mika grade, four-point Likert, and a hallucination taxonomy; PEMAT-P, DISCERN, and QUEST on pooled material). Readability used five indices. Between-model differences were tested with Friedman and Dunn-Bonferroni post-hoc tests; agreement and reproducibility used intraclass correlation coefficients (ICC).
Results:
All five readability indices differed between models (P < 0.001); none reached the recommended sixth-grade level. DeepSeek was most accessible (FRES 66.3) and ChatGPT least (FRES 17.7). On pooled-material instruments (inter-rater ICC 0.63-0.74), Claude scored highest for understandability (91.6%) and actionability (88.3%). Per-response clinical-quality measures were closely clustered, showed poor inter-rater agreement (ICC -0.01 to +0.07), and did not discriminate. Readability reproducibility across identical queries was moderate (FRES ICC 0.49-0.72).
Conclusions:
Clinical-quality scales did not reliably distinguish the four LLMs, whereas readability differed substantially; none met recommended readability standards, and output was only moderately reproducible. Model choice should be guided by readability and actionability, under clinician oversight.
Significance Of Study:
For a condition managed mainly through parent education, LLM selection should rest on readability and actionability rather than presumed clinical accuracy, with output used under clinician oversight.
Level Of Evidence:
V (cross-sectional comparative content analysis of AI-generated text; no human participants).