Related Experiment Video
Updated: Jul 3, 2025

A Computer-Based Platform for Aiding Clinicians in Eating Disorder Analysis and Diagnosis
Published on: May 10, 2022
Challenging the Chatbot: An Assessment of ChatGPT's Diagnoses and Recommendations for DBP Case Studies
Rachel Kim1, Alex Margolis, Joe Barile
1Division of Developmental and Behavioral Pediatrics, Steven and Alexandra Cohen's Children Medical Center of New York, Lake Success, NY.
Objective:
Chat Generative Pretrained Transformer-3.5 (ChatGPT) is a publicly available and free artificial intelligence chatbot that logs billions of visits per day; parents may rely on such tools for developmental and behavioral medical consultations. The objective of this study was to determine how ChatGPT evaluates developmental and behavioral pediatrics (DBP) case studies and makes recommendations and diagnoses.
Methods:
ChatGPT was asked to list treatment recommendations and a diagnosis for each of 97 DBP case studies. A panel of 3 DBP physicians evaluated ChatGPT's diagnostic accuracy and scored treatment recommendations on accuracy (5-point Likert scale) and completeness (3-point Likert scale). Physicians also assessed whether ChatGPT's treatment plan correctly addressed cultural and ethical issues for relevant cases. Scores were analyzed using Python, and descriptive statistics were computed.
Results:
The DBP panel agreed with ChatGPT's diagnosis for 66.2% of the case reports. The mean accuracy score of ChatGPT's treatment plan was deemed by physicians to be 4.6 (between entirely correct and more correct than incorrect), and the mean completeness was 2.6 (between complete and adequate). Physicians agreed that ChatGPT addressed relevant cultural issues in 10 out of the 11 appropriate cases and the ethical issues in the single ethical case.
Conclusion:
While ChatGPT can generate a comprehensive and adequate list of recommendations, the diagnosis accuracy rate is still low. Physicians must advise caution to patients when using such online sources.
More Related Videos
10:02Event Related Potentials ERPs and other EEG Based Methods for Extracting Biomarkers of Brain Dysfunction: Examples from Pediatric Attention Deficit/Hyperactivity Disorder ADHD
Published on: March 12, 2020
06:32Author Spotlight: Automated Deep Brain Stimulation for Parkinson's Disease - Exploring the Possibilities and Challenges of Home Monitoring
Published on: July 14, 2023
Related Concept Videos
Chronic Pancreatitis II: Collaborative Care
Assessment:
Methods of Documentation VI: Case Management Model
For example, a patient with a chronic...
Errors occurring during blood pressure monitoring
Several factors...
Pre-Procedural Guidelines for Assessing Blood Pressure
Special considerations while measuring blood pressure
Monitoring Both Arms:
Monitoring BP in both arms during the initial assessment is advisable, as the systolic value may differ by five to ten mm Hg between arms. For subsequent BP assessments, use the arm with the higher reading.
Diagnostic and Statistical Manual of Mental Disorders (DSM)