Evaluating large language models in pediatric fever management: a two-layer study.

Area of Science:

• Artificial Intelligence in Medicine
• Pediatric Health Informatics
• Natural Language Processing

Background:

• Pediatric fever is a common concern for parents, leading to frequent medical visits.
• Large language models (LLMs) show potential for medical communication but their use in pediatric fever is understudied.
• Evaluating LLM responses requires perspectives from both medical professionals and patient families.

Purpose of the Study:

• Compare the performance of four LLMs (ChatGPT3.5, ChatGPT4.0, YouChat, Perplexity) in answering pediatric fever questions.
• Assess how doctors and pediatric patients' relatives evaluate LLM-generated answers.
• Identify differences and similarities in LLM responses to complex pediatric fever queries.

Main Methods:

• Thirty pediatric fever-related questions were posed to four LLMs.
• Twenty doctors rated the LLM responses across four predefined dimensions.
• Pediatric relatives evaluated a subset of responses, with some revisited for deeper insights.
• Statistical analysis, including Tukey post-hoc tests, was employed to identify significant differences.

Main Results:

• Doctors rated ChatGPT3.5 and ChatGPT4.0 higher than YouChat and Perplexity across all dimensions.
• Accuracy was the highest-scoring dimension for all LLMs according to doctors.
• No significant differences in LLM response quality were found by pediatric relatives.
• Revisits with relatives indicated challenges in understanding and analyzing LLM outputs.

Conclusions:

• Internet-based LLMs (YouChat, Perplexity) did not enhance medical question-answering capabilities as anticipated.
• Patients' lack of medical knowledge and LLM answer structure hindered comprehension.
• Future LLM development for patient use should prioritize clear, central points and understandable language.