Comparative Evaluation of ChatGPT and ChatGLM Performance in Response to Common Queries on Pediatric Atopic...

Atopic dermatitis (AD) is a prevalent chronic and recurrent skin condition in children. Developing novel and standardized management strategies to control AD is urgently needed. Artificial intelligence technology-based large language models (LLMs), especially Chat Generative Pre-trained Transformer (ChatGPT) and Chat General Language Modeling (ChatGLM), show potential in generating appropriate responses to dialogue.

This study aims to assess the performance of ChatGPT-4 omni (ChatGPT-4o) and ChatGLM-4 in answering common queries about pediatric AD in a medical context. By screening popular inquiries from the AtopicDermatitis.net forum, we identified 102 key questions from parents of children with AD. Then, each question was input into both ChatGPT-4o and ChatGLM-4 to generate responses. Five senior dermatologists independently scored the reliability and clinical applicability of the responses. Finally, we compared the score distributions and performed a consistency analysis.

For both reliability and clinical applicability, ChatGPT-4o scored slightly better overall, ranging from 92.98% to 95.97% of the total maximum score, compared to ChatGLM-4, which ranged from 82.59% to 96.83%. However, there was no significant difference between them (p > 0.05). The consistency test indicated significant concordance among dermatologists (p < 0.05), with Kendall's coefficient of concordance above 0.40 in subgroups such as skin care, special manifestations, and treatment, demonstrating moderate consistency. They provide equivalent reliability and clinical applicability in answering queries about pediatric AD.

The quality of the two LLMs' responses matches that of dermatology professors, which demonstrates that LLMs can effectively recommend treatments, care, and management strategies for pediatric AD.