Related Experiment Video

Updated: Jul 8, 2025

A Computer-Based Platform for Aiding Clinicians in Eating Disorder Analysis and Diagnosis

A Computer-Based Platform for Aiding Clinicians in Eating Disorder Analysis and Diagnosis

Published on: May 10, 2022

Benchmarking the symptom-checking capabilities of ChatGPT for a broad range of diseases

Anjun Chen^1,2, Drake O Chen², Lu Tian³

¹Health Sciences, ELHS Institute, Palo Alto, CA 94306, United States.

Journal of the American Medical Informatics Association : JAMIA

|December 18, 2023

Abstract

Objective:

This study evaluates ChatGPT's symptom-checking accuracy across a broad range of diseases using the Mayo Clinic Symptom Checker patient service as a benchmark.

Methods:

We prompted ChatGPT with symptoms of 194 distinct diseases. By comparing its predictions with expectations, we calculated a relative comparative score (RCS) to gauge accuracy.

Results:

ChatGPT's GPT-4 model achieved an average RCS of 78.8%, outperforming the GPT-3.5-turbo by 10.5%. Some specialties scored above 90%.

Discussion:

The test set, although extensive, was not exhaustive. Future studies should include a more comprehensive disease spectrum.

Conclusion:

ChatGPT exhibits high accuracy in symptom checking for a broad range of diseases, showcasing its potential as a medical training tool in learning health systems to enhance care quality and address health disparities.

Keywords:

ChatGPT benchmarking learning health system medical training symptom checking

More Related Videos

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness

Published on: December 6, 2024

Author Spotlight: Therapeutic Benefit of Closed-Loop Deep Brain Stimulation in Depression Treatment

Author Spotlight: Therapeutic Benefit of Closed-Loop Deep Brain Stimulation in Depression Treatment

Published on: July 7, 2023

Related Concept Videos

Assessment of the Gastrointestinal System I: Subjective Data

Assessment of the Gastrointestinal System I: Subjective Data

Assessing the gastrointestinal (GI) system is a complex process that begins with collecting subjective data. This data, collected through patient interviews, provides crucial insights into the patient's health history, perception patterns, and lifestyle habits, all contributing significantly to GI health.
Health History
The initial step in assessing the GI system is obtaining a comprehensive health history. This includes inquiring about the patient's history or presence of problems...

Assessment of the Gastrointestinal System II: Health Perception Pattern

Assessment of the Gastrointestinal System II: Health Perception Pattern

Assessing the gastrointestinal (GI) system is a complex process that begins with collecting subjective data. This data, collected through patient interviews, provides crucial insights into the patient's health history, perception patterns, and lifestyle habits, all contributing significantly to GI health.
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:

Generalized Anxiety Disorder

Generalized Anxiety Disorder

Generalized Anxiety Disorder (GAD) is a chronic condition characterized by excessive and uncontrollable worry that persists for at least six months, significantly interfering with daily functioning. Unlike situational anxiety, which arises in response to specific stressors, GAD often occurs without a clear cause. Individuals may experience disproportionate worry about work, health, or relationships. For instance, a person might continuously fear poor health despite normal medical evaluations or...

Assessment of the Cardiovascular System I: Subjective Data

Assessment of the Cardiovascular System I: Subjective Data

A thorough health history and physical assessment are essential for identifying cardiovascular disease (CVD) symptoms and distinguishing them from other health issues.
Initial Enquiry
Ask the patient about their primary concern and thoroughly explore all reported symptoms.
Medical History
Investigate past illnesses affecting the cardiovascular system, such as angina, anemia, rheumatic fever, congenital heart disease, stroke, thrombophlebitis, dysrhythmias, varicosities
Inquire about symptoms...

Articles linked to this work by shared authors, journal, and citation graph.

Development of transient ischemic attack risk prediction model suitable for initializing a learning health system unit using electronic medical records.

BMC medical informatics and decision making·2024

Development of Lung Cancer Risk Prediction Machine Learning Models for Equitable Learning Health System: Retrospective Study.

JMIR AI·2024

Building Practical Risk Prediction Models for Nasopharyngeal Carcinoma Screening with Patient Graph Analysis and Machine Learning.

Cancer epidemiology, biomarkers & prevention : a publication of the American Association for Cancer Research, cosponsored by the American Society of Preventive Oncology·2022

The Generation of a Lung Cancer Health Factor Distribution Using Patient Graphs Constructed From Electronic Medical Records: Retrospective Study.

Journal of medical Internet research·2022

Simulation of a machine learning enabled learning health system for risk prediction using synthetic patient data.

Scientific reports·2022

Development and Validation of a Long-Term Incident Heart Failure Risk Model.

Circulation research·2021

See all related articles