Related Experiment Video
Updated: Sep 15, 2025
A Computerized Functional Skills Assessment and Training Program Targeting Technology Based Everyday Functional Skills
Published on: February 13, 2020
Evaluating the Agreement Between ChatGPT and the Clinical Competency Committee in Assigning ACGME Milestones for
Michael Partin1, Anthony B Dambro1, Roland Newman1
1Department of Family and Community Medicine, Penn State College of Medicine, Hershey, PA.
Background And Objectives:
Although artificial intelligence models have existed for decades, the demand for application of these tools within health care and especially medical education are exponentially expanding. Pressure is mounting to increase direct observation and faculty feedback for resident learners, which can create administrative burdens for a Clinical Competency Committee (CCC). This study aimed to assess the feasibility of utilizing a large language model (ChatGPT) in family medicine residency evaluation by comparing the agreement between ChatGPT and the CCC for the Accreditation Council for Graduate Medical Education (ACGME) family medicine milestone levels and examining potential biases in milestone assignment.
Methods:
Written faculty feedback for 24 residents from July 2022 to December 2022 at our institution was collated and de-identified. Using standardized prompts for each query, we used ChatGPT to assign milestone levels based on faculty feedback for 11 ACGME subcompetencies. We analyzed these levels for correlation and agreement between actual levels assigned by the CCC.
Results:
Using Pearson's correlation coefficient, we found an overall positive and strong correlation between ChatGPT and the CCC for competencies of patient care, medical knowledge, communication, and professionalism. We found no significant difference in correlation or mean difference in milestone level between male and female residents. No significant difference existed between residents with a high faculty feedback word count versus a low word count.
Conclusions:
This study demonstrates the feasibility for tools like ChatGPT to assist in the evaluation process of family medicine residents without apparent bias based on gender or word count.
More Related Videos
05:04Author Spotlight: Evaluating Clinicians' Adoption of Ultrasound-Guided Vascular Cannulation Through Simulation Training
Published on: August 9, 2024
05:56Implementation of Non-invasive Point of Care Transient Elastography for Evaluation of Liver Disease in Pediatric Populations with Cystic Fibrosis
Published on: August 29, 2025
Related Concept Videos
01:26Methods of Documentation II: POMR
01:19Standards of Care II
01:15Methods of Documentation VI: Case Management Model
For example, a patient with a chronic...
01:22Standards of Care I
01:26Methods of Documentation IV: Focus Charting
It typically involves three columns for recording information:
01:11Guidelines for Writing Outcome
Patient outcomes reflect the patient's response to the goal rather than what the nurse aims to achieve. Terminology should be observable and measurable to avoid the reader's interpretation. The desired outcome should be realistic and achievable in the designated care timeframe. Expected outcomes should align with adjunctive therapies. The outcome should enhance care...