A Comparative Evaluation of Three Large Language Models for Parent-Centered Questions About Anorexia Nervosa

Large language models (LLMs) are increasingly used to obtain health information, including guidance on child and adolescent mental health. In anorexia nervosa (AN), where early recognition and timely intervention are critical, the accuracy of AI-generated information available to parents may have important clinical implications. This study evaluated the performance of LLMs in responding to parent-oriented questions about AN.

A comparative model evaluation was conducted using three conversational AI systems: ChatGPT (GPT-4o), Google Gemini, and DeepSeek. Twenty questions representative of those frequently asked by parents of adolescents with AN were identified through online content exploration and expert review. Each question was submitted using standardized prompts in separate chat sessions. Responses were anonymized and independently evaluated by two board-certified child and adolescent psychiatrists across three dimensions: quality, usefulness, and reliability. Reproducibility was assessed through repeated queries in separate sessions.

All three models demonstrated generally high levels of reliability and overall informational performance. ChatGPT achieved the highest overall accuracy (≈92%) and reproducibility (≈90%), followed by Gemini (≈88% accuracy; ≈85% reproducibility) and DeepSeek (≈86% accuracy; ≈83% reproducibility). Domain-level analysis showed lower accuracy across models for diagnosis and clinical assessment questions. Qualitative error analysis indicated that the omission of clinically relevant information was the most common limitation, while DeepSeek produced more factual inaccuracies and Gemini more generalized guidance.

LLMs may provide broadly accurate preliminary information for parents seeking guidance about AN. However, persistent omissions and domain-specific variability highlight important limitations. AI-generated information should therefore be regarded as a complementary resource rather than a replacement for professional clinical guidance.