Related Experiment Video
Updated: Jun 4, 2025
Diagnosis of Musculus Gastrocnemius Tightness - Key Factors for the Clinical Examination
Published on: July 7, 2016
ChatGPT (GPT-4) versus doctors on complex cases of the Swedish family medicine specialist examination: an
Rasmus Arvidsson1,2, Ronny Gunnarsson3,4, Artin Entezarjou3
1General Practice / Family Medicine, School of Public Health and Community Medicine, Sahlgrenska Academy, University of Gothenburg Institute of Medicine, Gothenburg, Sweden rasmusalinks@gmail.com.
Background:
Recent breakthroughs in artificial intelligence research include the development of generative pretrained transformers (GPT). ChatGPT has been shown to perform well when answering several sets of medical multiple-choice questions. However, it has not been tested for writing free-text assessments of complex cases in primary care.
Objectives:
To compare the performance of ChatGPT, version GPT-4, with that of real doctors.
Design And Setting:
A blinded observational comparative study conducted in the Swedish primary care setting. Responses from GPT-4 and real doctors to cases from the Swedish family medicine specialist examination were scored by blinded reviewers, and the scores were compared.
Participants:
Anonymous responses from the Swedish family medicine specialist examination 2017-2022 were used.
Outcome Measures:
Primary: the mean difference in scores between GPT-4's responses and randomly selected responses by human doctors, as well as between GPT-4's responses and top-tier responses by human doctors. Secondary: the correlation between differences in response length and response score; the intraclass correlation coefficient between reviewers; and the percentage of maximum score achieved by each group in different subject categories.
Results:
The mean scores were 6.0, 7.2 and 4.5 for randomly selected doctor responses, top-tier doctor responses and GPT-4 responses, respectively, on a 10-point scale. The scores for the random doctor responses were, on average, 1.6 points higher than those of GPT-4 (p<0.001, 95% CI 0.9 to 2.2) and the top-tier doctor scores were, on average, 2.7 points higher than those of GPT-4 (p<0.001, 95 % CI 2.2 to 3.3). Following the release of GPT-4o, the experiment was repeated, although this time with only a single reviewer scoring the answers. In this follow-up, random doctor responses were scored 0.7 points higher than those of GPT-4o (p=0.044).
Conclusion:
In complex primary care cases, GPT-4 performs worse than human doctors taking the family medicine specialist examination. Future GPT-based chatbots may perform better, but comprehensive evaluations are needed before implementing chatbots for medical decision support in primary care.
More Related Videos
Related Concept Videos
01:29Assessment of the Gastrointestinal System II: Health Perception Pattern
Health Perception Patterns
Health perception patterns offer valuable insights into a patient's lifestyle habits and how they may impact their GI health. These patterns include:
01:30Serum Laboratory Studies, Stool Test, Breath Test
01:25Assessment of the Abdomen I: Inspection and Auscultation
The abdominal examination is a cornerstone of clinical medicine, serving as a critical tool in diagnosing various gastrointestinal (GI) diseases. It involves a systematic approach that includes inspection and auscultation, each with distinct yet complementary roles in assessing the abdomen. This article will delve into these two primary methods healthcare professionals use to examine the abdomen.
Inspection of the Abdomen
The first step in any abdominal examination is inspection....
01:17Assessment of the Gastrointestinal System I: Subjective Data
Health History
The initial step in assessing the GI system is obtaining a comprehensive health history. This includes inquiring about the patient's history or presence of problems...
01:26Pulmonary Tuberculosis IV
Several diagnostic approaches are used to detect TB. The conventional method is the Tuberculin Skin Test (TST), also known as the Mantoux test. However, this method has...
01:25Pulmonary Function Tests
Pulmonary Function Tests are crucial diagnostic tools for assessing respiratory function, particularly in patients with chronic respiratory disorders. They comprehensively evaluate lung volumes, ventilatory function, breathing mechanics, diffusion, and gas exchange. These tests help diagnose pulmonary diseases and play a significant role in monitoring disease progression, evaluating disability, and assessing response to therapy.
PFTs involve using a spirometer, a...