Evaluation of ChatGPT's reliability in answering questions about short stature and growth failure

To evaluate ChatGPT's performance in responding to questions related to short stature and growth failure, which are common concerns in clinical practice.

Answers provided by the GPT-4 and GPT-4o models to a 36-item questionnaire administered at three different time points (baseline, after 3 months, and after 1 year) were assessed. Each response was rated on a 0-6 Likert scale according to their agreement with those given by pediatric endocrinology specialists.

ChatGPT showed an overall satisfactory performance, with high agreement in conceptual questions and in respect of common causes, but lower accuracy in diagnostic and therapeutic topics that require more specialized knowledge. The mean score was 4.6 (SD = 1.3), with a mode of 6. Using a threshold of scores 3-6 as correct, 96.9% of responses were classified as satisfactory; applying stricter thresholds of scores 4-6 and 5-6 yielded rates of 77.2% and 46.3%, respectively. An improvement was observed after the update to GPT-4o, suggesting model evolution over time. However, there was a significant decrease in recommendations to seek medical care in the more recent model.

ChatGPT demonstrated high reliability and potential as a supportive tool for issues related to short stature; however, its lower accuracy in therapeutic topics and reduced emphasis on the importance of seeking professional medical advice highlight the need for further improvements to ensure its safe use in healthcare. Both professionals and lay users be aware of the limitations of AI and verify information through reliable medical sources.