Benchmarking Generative Artificial Intelligence Against Human Judgment in Eating Disorder Case Recognition and...

Generative AI is now used to access eating disorder (ED) information, yet the suitability of its outputs remains unclear. Using two vignettes, we evaluated generative AI's ability to accurately identify ED presentations and recommend appropriate care, while benchmarking its performance against published responses from human samples.

ChatGPT-5.4 was presented with two vignettes and prompted with questions related to problem identification and treatment recommendation. Vignette one depicted a restrictive ED consistent with anorexia nervosa or atypical anorexia nervosa, with body weight varying across low-, average-, and higher-weight conditions. Vignette two depicted a person with binge-eating disorder (BED). Responses across 20 prompt administrations per vignette were compared against benchmark data from clinicians for vignette one and community participants for vignette two.

For vignette one, ChatGPT identified an ED or possible ED in 100% of responses for the low- and average-weight conditions and 90% for the higher-weight condition, compared with 47%, 21%, and 16% among human clinicians, respectively. It recommended specialized ED treatment 100% of the time across all three weight conditions, compared with 35%, 19%, and 17% of clinicians, respectively. For vignette two, ChatGPT correctly identified BED as the problem in 100% of prompt repetitions (vs. 59% in community participants) and more consistently endorsed evidence-based treatments than community participants (100% vs. 50%). Supplementary analyses confirmed similar results when prompting Gemini-3 and Claude-4.5.

Generative AI can accurately identify ED presentations and recommend suitable care in structured scenarios, while showing little evidence of common biases observed in human samples.